Unannounced or announced periodic hospital surveys: a study protocol for a nationwide cluster-randomized controlled trial

Introduction

Accreditation by external audit to validate that health care organizations’ operations and practices satisfy agreed-upon high-quality service criteria is conducted in more than 70 countries.^1,2 In 2003, the Danish government and the Danish Regions (responsible for all public hospitals) decided to implement a nationwide model of quality in health care which involved formal accreditation of all Danish public hospitals. In 2007, a national Institute for Quality and Accreditation in Healthcare (IKAS) was formed to develop the Danish Healthcare Quality Program (DDKM) and since 2010, all public hospitals have been accredited based on announced organization-wide surveys which are conducted every third year (including a periodic midterm survey). Other types of health care institutions are now also included in the program and accredited on a regular basis (eg, private hospitals covered by the extended free hospital choice, pharmacies, and nursing homes), and a plan for accreditation in general practices was negotiated in 2014.^3–5

The DDKM is based on announced surveys, and the procedure of preannouncing the dates for all surveys has been criticized for creating an “arranged reality” and for not reflecting the daily work with quality of care. This criticism is not unique to the Danish setting, but has been raised in a number of countries that operate accreditation systems based upon announced surveys.^6,7 Another possible negative effect of announcing the surveys is the considerable amount of time and human resources that are spent to prepare for the announced survey. This implies that less time is available for patient care in the period of preparations.⁶ It has been proposed that the implementation of unannounced accreditation surveys may be instrumental in alleviating these problems. An unannounced survey is an external visit paid to an organization without prior notice of when the visit will take place.²

Unannounced surveys have been used for several years by the Joint Commission in the United States and the Aged Care Standards and Accreditation Agency in Australia, but, to our knowledge, no peer-reviewed literature has been published on the experiences of changing the procedure from announced to unannounced surveys.^2,7,8 Only one project, launched by the Center for Clinical Governance Research (CCGR) in Australia in 2012, encompassed an empirical test of short-notice surveys in two accreditation programs. This study was conducted in a paired design where short-notice surveys were compared with the most recent advance notification survey. The study found that use of the short-notice survey approach to the rating of organizational performance was less successful than the advance-notification survey (5% significance level, P=0.044).^6,9

The present trial was designed by IKAS and the Danish Center for Healthcare Improvements (DCHI) to inform a decision of whether or not to implement unannounced accreditation surveys in the third version of the DDKM in 2016. In September 2013, the IKAS Board of Directors decided to approve and finance the project, and it is expected that the results from the present trial will be available for the board’s decision in autumn, 2015.

To provide the best possible basis for political decision making, the present study was designed as a nationwide cluster-randomized controlled trial (C-RCT) including general hospitals, university hospitals, and psychiatric hospitals from all five regions in Denmark. The objective of this trial is to evaluate the effect of unannounced hospital surveys at cluster level based on findings of a survey embracing an abbreviated set of the national accreditation standards and performance indicators from the DDKM version 2. We hypothesize that unannounced surveys produce less successful measures of organizational performance than announced surveys. This study defines less successful as less compliance with the included accreditation standards and performance indicators.

Materials and methods

The DDKM version 2

The Danish health care sector consists predominantly of public hospitals organized into five regions, a municipal citizen-centered health care with public nursing homes and home care, and family doctors and specialists organized as a private sector.¹⁰ Public hospitals are being accredited according to the DDKM version 2. The aim of the DDKM is “to promote the quality of the continuity of care, promote the development of the clinical, organizational and the patient-experienced quality, and to visualize quality in the health care system”.¹¹ The DDKM version 2 comprises a set of accreditation standards (n=82) and performance indicators (n=473), integrated into three themes: organizational standards, general standards of continuity of care, and disease-specific standards. The methodology of the DDKM is based on the Plan-Do-Study-Act cycle.^11,12 “Plan” is addressed to the hospitals’ responsibility to write guidelines describing how quality goals in a specific accreditation standard can be fulfilled; “Do” is addressed to the hospitals’ responsibility to ensure implementation of the guidelines; “Study” is addressed to the hospitals’ responsibility to ensure monitoring of the quality of processes, structures, and health benefits; and “Act” is addressed to the hospitals’ responsibility to assess the results of data monitoring as well as ensuring initiatives for quality improvement. All four categories are assessed with predetermined performance indicators.^11,13

The DDKM has a 3-year cycle that includes two onsite visits. The first visit is an organization-wide survey examining both mandatory and non-mandatory accreditation standards. The second visit is a periodic survey 1.5 years ±6 months after the organization-wide survey is conducted. The purpose of the second visit is to ensure ongoing compliance with the accreditation standards and performance indicators.^5,11,13

The assessment of performance indicators during organization-wide surveys are based on a four-level scale (“completely fulfilled”, “significantly fulfilled”, “to some degree fulfilled”, and “not fulfilled”). To gain the status “accredited”, the highest level of performance must be reached for the mandatory standards (eight critical patient-safety standards), and the highest or the second highest level of performance must be reached for the non-mandatory standards. Second highest level meaning that the few flaws is not a considerable part of the indicators requirement, assessed by the surveyor. Hospitals can receive the status “accredited”, “accredited with remarks”, “conditionally accredited”, or “not accredited” after an accreditation survey. If the hospital is “accredited with remarks” or “conditionally accredited”, it must improve and document the inadequacies to receive a final status as “accredited”. If the hospital is not accredited, a new organization-wide accreditation survey is required, and the assessment process starts over again.

Study design

This trial is a nationwide C-RCT with a test period of 9 months (August 2014 to May 2015). The hospitals included in this trial are defined as the clusters because the wards within each hospital are clustered. A standardized tracer method was designed specifically for the purpose of this trial to ensure standardization of each survey execution. The only difference between the two clusters is whether a hospital received the survey on an announced or unannounced basis. The survey process proceeded over 2 days and was followed by a “quality and management interview” on day 3. The “quality and management interview” aimed at providing feedback about the 2-day survey process and reflects the management’s work with organizational quality development. The 2-day survey process was planned to be conducted over a short period of time for the majority of the surveys. The period was short to avoid the hospitals being able to inform each other about the trial start-up.

Recruitment of hospitals

On April 1, 2014, all public somatic and psychiatric hospitals in Denmark (n=30) received a letter from IKAS inviting them to participate in the trial. The invitation contained information about the survey’s 2-day tracer activity and the planned “quality and management” interview. The hospitals were also informed that the results of the survey would be available for the hospitals only.

Ten of the invited hospitals were to have a periodic survey within the study period (August 1, 2014 to May 1, 2015) as they were already covered by the accreditation process and accreditation was hence mandatory. They accepted replacing their periodic survey with a trial survey. Another 20 hospitals were contacted and invited to volunteer their participation, whereby they would receive an extra survey on top of the surveys already scheduled as part of the regular 3-year survey process. Seven hospitals rejected participation (six hospitals from the Capital Region of Denmark and one from Region Zealand) (Figure 1).

Figure 1 Participant flow diagram representing the number of clusters that were randomly assigned to either announced or unannounced trial surveys.

Twenty-three public hospitals (77%) were included in this study, constituting the clusters: three university hospitals, five psychiatric hospitals, and 15 general hospitals. All five Danish regions are represented in this trial.

Recruitment of surveyors

All nine surveyors were professional clinicians with at least 3 years of experience as a surveyor. They were chosen and educated by IKAS to conduct surveys for this particular trial. Restrictions were made concerning the number of surveyors for this trial to ensure consistency and homogeneity of the data collection. In addition, a small group of surveyors was more likely than a large group to comply with confidentiality of the study design.

Randomization of hospitals and surveyors

The random allocation sequence was conducted by DCHI and performed in Excel, assigning eleven hospitals to receive announced surveys (control group) and 12 hospitals to receive unannounced surveys (intervention group) (Figure 1). The different strata represent university hospitals, general hospitals, and psychiatric hospitals. These strata represent diversity in both size and the core output of treatment, respectively referring to diversity in somatic and psychiatric hospitals. The participating hospitals were therefore randomized in blocks, ensuring a balance of strata allocated to both groups. A restriction was made prior to the randomization to ensure that each region received both announced and unannounced surveys. The dates of the surveys were arranged from the beginning and then randomly assigned to each hospital within the intervention and control groups.

The hospitals were blinded throughout the trial in the sense that all hospitals believed that they received an announced survey. This was done by announcing a date for the quality and management interview, which is usually undertaken straight after the completion of a survey process. On day three the hospitals were also informed to act in the same way as they usually would for a regular, announced survey. The hospitals randomized to receive announced surveys were informed 1 week before the planned interview to make the practical arrangements for the forthcoming survey, eg, the hospitals were asked to find employees who could assist the surveyors on the specific locations where the survey would take place. The hospitals that were randomized to receive unannounced surveys had their surveys performed approximately 3 months earlier than announced.

The following restrictions concerning the randomization of surveyors applied: a surveyor group had to include two surveyors – one medical doctor and one nurse. The team had to include surveyors from two different regions. Neither of the surveyors was allowed to survey hospitals from his or her own region. All possible combinations of surveyors had to be utilized except in case of practical impossibilities. Finally, every surveyor had to participate in both announced and unannounced surveys. The surveyors were blinded for as long as possible, but if they were to survey a hospital in the control group, they were informed about this 1 week before the survey, due to practical arrangements as for a regular survey.

The randomization was performed and held confidential in a locked fireproof cabinet to which only DCHI had access. This ensured an additional blinding of IKAS. For practical and logistic purposes, IKAS received the randomization of hospitals and surveyors, but as late in the process as possible.

Accreditation standards and performance indicators

The data collection was based on an abbreviated version of the accreditation standards (48 accreditation standards and 113 indicators) from the second version of the DDKM. The included standards comprised organizational standards (n=16), general continuity of care standards (n=30), and disease-specific standards (n=2). As a normal accreditation survey, there are four levels (level 1–4) of performance indicators for each standard which are used for assessment purposes in this trial. Included in this trial were: step 1 indicators “Plan” (n=4), step 2 indicators “Do” (n=102), step 3 indicators “Study” (n=5), and step 4 indicators “Act” (n=2).

Surveyors were to apply the tracer method to evaluate clinical practice, for which reason step 2 indicators for specific clinical standards are overrepresented in this trial.

Outcomes

The primary outcome of this trial is organizational performance according to pre-chosen standards and performance indicators. It was expected that approximately 700 measurements per hospital would be applicable for data analysis. This assumption was based on a calculation of the 113 pre-chosen performance indicators included for assessment, measurements from two surveyors per hospital, and a mean of three wards being assessed per hospital. The evaluated performance indicators (n=113) were assigned one out of four predetermined phrases reflecting organizational performance, enabling a conversion of data into categorical ordinal variables. The predetermined phrases are worded as follows: “consistent implementation” (variable 1), “consistent implementation with single deviations” (variable 2), “weak implementation” (variable 3), and “missing implementation” (variable 4). All findings were accumulated on each performance indicator. Outcomes will be analyzed and reported at both hospital (cluster) level and indicator level (population). We obtained two sets of measurements at each level, one set from each surveyor.

Sample size

We conducted a sample size calculation allowing for the cluster randomization using the “clustersampsi” procedure downloaded to STATA 13 (StataCorp LP, College Station, TX, USA). Based on previous surveys, we expected to find 5% not consistently implemented standards in the control group and an estimated 10% not consistently implemented standards in the intervention group. We used an intra-intraclass correlation coefficient of 0.01, eleven hospitals per arm, 700 measurements per hospital, and a significance level of 0.05, thus obtaining a power of 98% to detect a minimum absolute risk difference of 5 percentage points.

Statistical methods

The statistical analyses were divided into two parts: bivariate analyses of the findings by the 113 performance indicators and a pooled analysis of all indicators assessing the common difference between findings in the two trial arms. In the pooled analyses, an adjustment will be made for the indicator levels as fixed effects. Both the bivariate and pooled analyses used binomial regression, bootstrapping the standard error of the parameter of interest through resampling with replacement of the 23 clusters, ie, the 23 hospitals. The bootstrap resampling of the data on hospitals was repeated 100 times. A resampling of the 23 hospital clusters were performed to avoid exaggerating the variation within a hospital due to clustering. The dichotomous outcome of the binomial regression was defined as “one” if assessment of a performance indicator was “consistently implemented”; otherwise the outcome was coded as “zero”. Risk difference is applied to the association measure in the binomial regression analyses because the absolute difference between the frequency of consistent findings in the unannounced arm and the frequency of consistent findings in the announced arm seemed the most appropriate choice. A two-sided significance level of 0.05 applies for all analyses. All analyses were performed in STATA 13 and will be presented according to the extensions of the CONSORT statement for cluster-randomized controlled trials.¹⁴

Trial status

The collection of data is closed and the authors are currently analysing data.

Ethics

According to the Act on Research Ethics Review of Health Research Project, act number 593 of July 14, 2011, section 14, it was decided by the Central Denmark Region Committees on Health Research Ethics that this study is not a health research study and should not be notified to the committees.

Discussion

This study is the first C-RCT of unannounced versus announced periodic hospital surveys in the Danish health care setting. It is a part of a nationwide research project combining three studies: the present, nationwide survey study, and a qualitative study including observation and interviews. The survey, delivered to more than 17,000 health care professionals, investigates the attitudes toward unannounced surveys, the DDKM, and the national accreditation program. The qualitative study investigates the experiences and attitudes toward unannounced surveys of hospital staff and surveyors. Up until now, only short-notice surveys have been empirically evaluated. In the study of short-notice surveys, participating organizations were given a 2-day notice before the survey and a version of the accreditation standards to be examined during the survey.^6,15 It could be argued that the present trial, consisting of unannounced surveys with no preparation possibilities, is unsuitable for assessment of accreditation standards and performance indicators at the organizational level due, among other things, to planning deficiencies, eg, participating hospitals having insufficient time to prepare and procure information, insufficient time to dispose pathfinders for the accomplishment of the survey process, and insufficient time to arrange the “quality and management interview”. On the other hand, we argue that the 2-day prior notice would involve a risk of disclosing the randomization due to the well-established network existing between the Danish hospitals.

Six hospitals from the Capital Region of Denmark and one hospital from Region Zealand refused to participate in this trial. One could argue that a refusal from these specific hospitals was due to their next organization-wide accreditation survey being conducted immediately after the trial period.

The included hospitals were informed that any possibly negative findings would not be available for the public. Thus, there was a risk that the hospitals in the group of the announced survey would use the trial as an opportunity for them to test their frontline personnel in unannounced surveys in a setting carrying only little risk of political consequences. Furthermore, there was a risk that the group receiving announced surveys might not prepare for the survey because they may not believe that the announced dates were true since all hospitals were given dates for a periodic survey as a part of the trial design. Accordingly, all hospitals could have acted similarly, which may imply that there would be only a slight or no difference between the two randomization groups. If this were the case, the effects of introducing unannounced surveys in the Danish hospital sector may be said to have been underestimated in this study.

It was decided that a blinding of the hospitals, in the sense that all hospitals believed that they received an announced survey, was necessary to ensure that the control group would prepare themselves as they usually would for an accreditation survey. Without the blinding, there was a risk that the intervention group would prepare for the survey even though they were supposed not to.

The standardized tracer tool and the time constraint required for the purpose of this trial could have caused surveyors to grade performance indicators wrong, by either rejecting singular findings (false negative) or the risk of not detecting an ordinary finding (false positive).

In addition, it is unusual to conduct randomized controlled trials at the organizational level, because these settings have several uncontrollable variables. This design was, for example, unable to control how the hospital management at all levels would decide to involve the remaining hospital staff in the design of this trial.¹⁶

The large scale of this study, covering hospitals from every region of Denmark and more than three quarters of the Danish hospitals, will allow us to generalize the results to the national level. Whether these results could be generalized internationally cannot be determined. It could be argued that the quality of the accreditation standards, the survey education, and the survey processes are largely comparable between the countries having accreditation organizations being accredited by the global organization the International Society for Quality in Health Care (ISQua). ISQua ensures that accreditation is conducted based on the same principles (ISQua’s meta-standards) across countries.¹³

Existing literature will be used as validation material for the trial results.

The findings of this trial will be disseminated through peer-reviewed journals and through national and international conferences and will be utilized for political decision making within the health care sector for the future national accreditation programs.

In closing, this study facilitates a validation of the effect of unannounced periodic hospital surveys and thereby addresses whether the conventional, announced survey can be considered an “arranged reality”. Future studies investigating the effect of unannounced surveys could preferably investigate the patient experience for benchmarking purposes of announced and unannounced surveys. In addition, accreditation programs have never been evaluated in a health economic respect, which could provide important information about the cost-effectiveness of unannounced surveys compared with the conventional announced surveys. However, the international value of a national cost-effectiveness study is questionable due to limited generalizability. CCGR has presented a protocol article describing a future study to conduct a cost-effectiveness analysis of different accreditation models, with, however, no inclusion of unannounced surveys.⁹

Acknowledgments

This C-RCT is funded by IKAS. The authors would like to thank the hospitals for their participation. In addition, thanks go to the surveyors for their engagement and great work in preparing and conducting the surveys. In closing, thanks go to IKAS for contributing to the conception of the study and the overall study design. The abstract in this paper is presented according to the CONSORT Extension for Abstracts checklist. This trial is registered at ClinicalTrials.gov (NCT02348567).

Author contributions

All authors contributed toward data analysis, drafting and revising the paper and agree to be accountable for all aspects of the work.

Disclosure

The authors report no conflicts of interest in this work.

---

References

1.	Jaafaripooyan E, Agrizzi D, Akbari-Haghighi F. Healthcare accreditation systems: further perspectives on performance measures. Int J Qual Health Care. 2011;23:645–656.
2.	Greenfield D, Braithwaite J. Developing the evidence base for accreditation of healthcare organisations: a call for transparency and innovation. Qual Saf Health Care. 2009;18:162–163.
3.	Indenrigs- og Sundhedsministeriet, Amtsrådsforeningen, Hovedstadens Sygehusfællesskab. Samarbejdsaftale Om Etablering Af En Fælles Model for Kvalitetsvurdering. Available at: http://www.ikas.dk/Admin/Public/Download.aspx?file=Files%2FFiler%2FDiverse+filer%2FSamarbejdsaftale-DDKM-2004.pdf. Accessed 27 July, 2015. 2004: Page 11. Danish.
4.	Forhistorien [webpage on the Internet]. Aarhus: Institut for Kvalitet og Akkreditering i Sundhedsvæsenet (IKAS) [updated January 28, 2014]. Available from: http://www.ikas.dk/DDKM/Visionen/Forhistorie.aspx. Accessed July 27, 2015. Danish.
5.	Håndbog i DDKM for sygehuse, 2. version, [A handbook for The Danish Healthcare Quality Programme (DDKM) version 2 for hospitals. Aarhus: Institute for Quality and Accreditation in Healthcare (IKAS). Available from: [http://www.ikas.dk/Sundhedsfaglig/Sygehuse/H%C3%A5ndbog-i-DDKM-for-sygehuse--2.-version.aspx. Archived at: http://www.webcitation.org/6WSjuqMz5]. Danish. Accessed 27 July 2015.
6.	Greenfield D, Moldovan M, Westbrook M, et al. An empirical test of short notice surveys in two accreditation programmes. Int J Qual Health Care. 2012;24:65–71.
7.	Greenfield D, Travaglia J, Braithwaite J and Pawsey M: UNANNOUNCED SURVEYS AND TRACER METHODOLOGY: LITERATURE REVIEW. The University of New South Wales, Centre for Clinical Governance Research. A Report for the Australian Accreditation Research Network: Examining future Health Care Accreditation Research 2007:1–30. ISBN: 9780733425462.
8.	Greenfield D, Braithwaite J. A Review of Health Sector Accreditation Research Literature. 2008:20(3):172–183.
9.	Braithwaite J, Westbrook J, Johnston B, et al. Strengthening organizational performance through accreditation research-a framework for twelve interrelated studies: the ACCREDIT project study protocol. BMC Res Notes. 2011;4:390.
10.	Pedersen KM: Sundhedspolitik, [Health Policy]. Syddansk Universitetsforlag, 2005:718 pages. Danish. ISBN: 9788778389794.
11.	Institut for kvalitet og akkreditering i sundhedsvæsenet: Akkrediteringsstandarder for Sygehuse, 2. Version. Aarhus: Institute for Quality and Accreditation in Healthcare (IKAS) 2012:213 pages. Available from: http://www.ikas.dk/Sundhedsfaglig/Sygehuse/2.-version.-Akkrediteringsstandarder-for-sygehuse.aspx. Accessed: 10 July 2015. Danish.
12.	Taylor MJ, McNicholas C, Nicolay C, Darzi A, Bell D, Reed JE. Systematic review of the application of the plan-do-study-act method to improve quality in healthcare. BMJ Qual Saf. 2014;23:290–298.
13.	Mainz J, Bartels P, Bek T, Pedersen K, Krøll V, Rhode P. Kvalitetsudvikling I Praksis. Munksgaard Danmark; 2011: Page 427. ISBN: 9788762807143.
14.	Campell M, Piaggio G, Elburne D, DG. A: Consort 2010 statement: extension to cluster randomised trials. BMJ 2012, 345.
15.	The Australian Council on Healthcare Standards In partnership with Australian General Practice Accreditation Limited (AGPAL): Final Report Piloting Innovative Accreditation Methodologies: Short Notice Surveys. 2009(May 2009):117 pages. Available from: http://www.safetyandquality.gov.au/wp-content/uploads/2012/01/Piloting-of-Short-Notice-Surveys-in-Accreditation-%23U2013-Completed-May-2009.pdf. Accessed: 27 July, 2015..
16.	Campbell M, Fitzpatrick R, Haines A, et al. Framework for design and evaluation of complex interventions to improve health. BMJ. 2000;321:694–696.