Assessing the Validity and Cover Rate of the National Swedish Hernia Register

Introduction

Randomized controlled trials (RCT) are considered to be the gold standard when studying the effect of medicines or surgical procedures. A well performed randomization ensures the intervention to be the only difference between the groups resulting in high internal validity.¹ However, problems may arise if surgical methods are compared. Patient- and surgeon-related factors may influence outcome. The external validity and hence the generalizability can also be a problem since a RCT studies the efficacy of an intervention, mostly but not consistently, performed under more optimal circumstances by experts and for selected patients.

In contrast a national quality register with high coverage presents the effectiveness of an intervention or the result of an intervention when implemented by many surgeons, in different settings and for unselected patients within the country. Outcome thus reflects routine practice. Register studies can be a complement to a RCT when assessing the results of a surgical procedure when implemented in routine care.² Furthermore, because of the power and magnitude of registered procedures and the simple endpoints, register studies can be used when studying rare events or events that require a long follow-up time. When correct, data from medical registers provide a unique possibility to improve quality of care and to facilitate patient-related research.^3,4

An essential prerequisite for a national register is high coverage and high validity of data. The higher the coverage the higher the generalizability from research based upon data from the register since the risk of skewed patient selection decreases. A fundamental issue is the validity of input data. Care must be taken when interpreting registry data since incorrect or missing data reduces the credibility of the register. These factors are crucial for the external validity and generalizability of conclusions reached in studies involving registry data.

In surgery, several registers have been established in recent decades.^5–13 The Swedish Hernia Register (SHR), founded in 1992, was the first surgical national quality register in the world. The SHR has an almost complete coverage of Swedish groin hernia surgery and has grown to include detailed information on more than 350,000 operations. The primary aim is to assess the quality of groin hernia surgery in Sweden, to stimulate improvement at the participating units and provide the basis for scientific research.

The aim of this study was to evaluate a national hernia registry in terms of data validity and coverage.

Method

In this cross-sectional observational study registered data from the SHR was compared with medical records including hospital notes regarding the hernia operation. Physicians at almost all hernia operating units participate in registering hernia operations in SHR. At each participating unit a physician and secretary are accountable for the registration process. Each patient is followed in the register until death, emigration, or a reoperation of a hernia in the same groin making it possible to deduce a cumulative incidence of reoperation for recurrence of groin hernia.¹⁴ All Swedish citizens have a unique personal identification number which makes it possible to follow individuals over time regardless of where in Sweden they live and seek medical care.¹⁵

Variables registered include hernia anatomy, handling of nerves, method of repair and anesthesia used. Patient-related variables such as sex, age, BMI and comorbidity are also included. Participation is voluntary for the patient and can at any time be withdrawn in accordance with the Swedish Patient Data Act.¹⁶ To assess both completeness and validity a continuing validation has been performed during the time of the register.¹⁷

Validation

Since the start of SHR an annual validity control of randomly selected hospitals has been carried out, and since 2013 in a more standardized form to allow a systematic data collection and evaluation. Specially trained independent nurses familiar with source data verification and groin hernia surgery were tasked to validate the registered data annually. Data prospectively registered in the register in 2012 were validated in 2013 and so forth. Each year, 10% of participating clinics were randomly selected in a specific region of Sweden, ensuring a systematic validation of all regions from north to south. Two validators visited the selected hospitals. A summary list of all hernia operations performed during the year was provided as well as formal access for the validator to the relevant medical records. Register data were validated against medical records regarding coverage of all operations in the hospital and accuracy of data registered. Data that in any way differed from the medical record were considered as a faulty variable. Any complication, however small, had to be registered otherwise the variables regarding postoperative complications would be considered faulty. The validators recorded operations not found in the SHR and errors of data registered. 50 registered operations were randomly selected at each unit and scrutinized regarding 27 variables for each registered operation. Some variables determined to be of extra importance were validated at all units while other variables varied randomly each year. After validation, the clinic received a report of the results.

Missing Data

To complete a registration of an operation in SHR information on all the variables have to be filled in. Therefore for every registration there are data available on all variables. There are no missing data in the SHR only incorrect data or a missed registration altogether.

The Regional Ethical Board at Gothenburg University, Sweden approved this national study (EPN 417-17).

Results

The cover rate of the SHR was examined with results from 2014 to 2018 and included 44 hospitals as shown in Table 1. The cover rate of hernia operations performed at the hospitals and registered in the SHR ranged from 95% to 99% and in mean 97%.

Table 1 Cover Rate of the Swedish Hernia Register from 2014 to 2018

Fifty-five participating hospitals were randomly selected for validation of data from 2013 to 2018 within Sweden. The number of hospitals validated each year ranged from 15 to 6 and in total, 2740 patient records were examined as shown in Table 2. The number of examined cases each year ranged from 762 to 300 depending on the year of validation and all in all, 73,764 variables have been compared with the medical records. The proportion of registrations with more than one error was 9% and varied from 6% to 15% between different years of validation. Total proportion of correct variables was 98% in total ranging from 98% to 99%.

Table 2 Data Quality in the Swedish Hernia Register from 2013 to 2018

The frequency of errors in 10 variables is presented in Table 3. The most common error in the registration was a faulty ASA score and this represented 17% of all errors. If there was a mismatch between the ASA classified by the anesthesiologist in the medical records and the data in the register the variable was considered incorrect. The second most common error was the date when the patient was put in the waiting list for surgery. The third most common error was postoperative complications within 30 days. A failure to register a complication noted in the medical record during the first 30 days after the operation was considered as an incorrect variable. The nature of the incorrect variable is not noted consistently in the validation. However, a frequent mistake was the failure to report urinary retention resulting in catheterization as a postoperative complication.

Table 3 The Frequency of Errors in 10 Variables

Discussion

An essential prerequisite for the results from a register-based study to be credible is that registered data have high validity and that the register has a high degree of coverage.

Independent reviewers have found the cover rate and validity of the SHR to be of good quality. Between the years 2013 to 2018, 73,764 variables were examined with only 2% inaccuracy. The errors in registration were also in most cases considered to be minor, not affecting the results of pooled data used for scientific studies.

From 2013 to 2018 the number of registered operations in SHR ranged from 15,616 to 16,577 with a mean of 15,946.¹⁸ The number of cases audited regarding cover rate represents a mean of 15% of all cases. The number of cases audited regarding data quality represents a mean of 3%. A large number of hospitals and patient data was examined and 2740 medical records have been compared with corresponding data registered in the SHR. This represents hospitals in all parts of Sweden and over several years in order to provide a representative selection for this analysis of variable audit. Furthermore, the validation process was performed with four specially educated nurses, with a strictly defined protocol in order for the validation process to be as identical as possible in all of the validated units. To our knowledge SHR have the largest validation process of a hernia register and provides a unique basis for assessing the quality of the data registered.

The study shows a proportion of 9% registered operations with more than one error indicating that the validity of the register can be improved further. The variations over time regarding cases with more than one error illustrates the importance of continuous quality control and follow up of the registration process. The validators have also functioned as educators showing the responsible personnel at the local hospitals how to avoid systematic errors and were able to answer their questions.

Despite a clear definition of ASA score as defined by the anesthesiologist, ASA score was still the most common error in the registration. The second most common error was the date that the patient was put on the waiting list for surgery. One explanation for this might be the fact that is easy to mistake this date with the referral date or the date of the first visit. Incorrect registration of postoperative complications was the third most common error and is of great importance. Failure to report complications might give a bias regarding the safety and efficiency of an intervention and units that avoid or fail to register their complications show a falsely improved outcome. In the validation the specific nature of the complication is not systematically recorded but is in many cases noted. This gives the possibility to deduce that the missed complications were largely due to missing information regarding catheterization after postoperative urine retention. Incorrect registration of complications must be kept at a minimum and efforts to reduce its frequency should be continuous.

There are over 100 national medical registers in Sweden and their cover rate and validity varies greatly.¹⁹ Different approaches to validation are evident as the registers are different in design and complexity. Validation can be done in several ways, for example with re-extraction of data or by cross-reference to other registers^20–22 for example the Patient Administrative System (PAS) that includes the ICD coding on all Swedish hospitals. This makes it possible to detect missing data from the register. High demands are thus placed on the register used as cross-reference for the validation to be of good quality. When using validation with re-extraction of data from the medical charts this can be done with one or several extractions over time. The cases can be selected or randomized.^23,24 Comparing registers is difficult since they were quite a lot in design and scope. Even so, the cover rate and validity of the SHR could be considered good when compared with other national registers.^19,23 Similarly it is difficult to compare different hernia registers. Mainly because the type of audit that this study represents is rare in other hernia registers. The difference that can be suggested is that SHR might have a better cover rate compared with other hernia registers. Another difference is that SHR and the Danish hernia database is publicly funded in contrast to other hernia registers.²⁴

A surgical quality register with national coverage including all operations provides a unique opportunity to fully evaluate the effect of an intervention, when implemented by all surgeons and type of units and without excluding patients by age, gender or comorbidity. This is of great importance since an operation that is safe and effective in one category of patients might not be suitable or safe for all patients. One example is hernia repair in women where a laparoscopic approach is recommended to minimize risk of femoral recurrences and reduce reoperation rate.²⁵ These differences can only be detected with a large patient base because of the differences in cumulative incidence of inguinal hernia between men and women and because of the rate of recurrence after hernia surgery. Another example is the detection of a type of mesh used in hernia surgery associated with an increased risk of recurrence using registry data.²⁶ Both of these examples illustrate the importance and clinical implications of registers with high cover rate and high validity. A high cover rate also ensures that not only the successful operations are registered and evaluated. This in turn provides the basis for comparing interventions in clinical practice.

Several measures are taken in order to maintain and improve both internal and external validity. To ensure high external validity of the SHR the annual review at randomly selected hospitals is performed by an independent reviewer to compare register data with medical records. In order to promote high internal validity, the unit’s own responsibility to deliver accurate data is stressed. Other measures to increase internal validity is that data are registered online in direct connection with the operation and that the registration is cross-referenced with internal data lists to decrease risk of missed registrations.

In conclusion, this validation of the Swedish Hernia Register shows high cover rate and good quality of data. A well-validated national surgical register with a high coverage provides a unique opportunity to serve as a basis for continuous quality development and population-based research reflecting routine surgery. Conversely, a registry with missed and incorrect data can provide false and unreliable outcomes.