Application of the Adaptive Validation Substudy Design to Colorectal Cancer Recurrence

Introduction

Colorectal cancer (CRC) is the third most commonly diagnosed malignancy and the third leading cause of cancer-related mortality among men and women globally.¹ Improvements in diagnostic workup, including screening and in surgical techniques and medical treatments, have contributed to an increase in the survival rates for CRC patients.^2,3 Current survival estimates are largely dependent on the stage at diagnosis, with estimated 5-year survival proportions of 90%, 70%, and 10% for stage I–II, stage III, and metastatic disease, respectively.⁴

A cancer recurrence is broadly defined as a return of the original cancer following an initial disease-free period.^5,6 A cancer recurrence can happen locally (at or near the original site), regionally (in surrounding lymph nodes), or distally (at a distant anatomical site).⁷ Among individuals diagnosed with CRC, the 5-year risk of recurrence is between 20% and 50%, depending on stage and other tumor characteristics at diagnosis.^7–11 Treatments for local or regional (stage I–III) disease include surgery alone or combinations of surgery, chemotherapy, and radiation therapy, with the intent of curing the patient of their disease.¹² Following curative treatment and an apparent disease-free period, a recurrence is a necessary preceding condition before CRC-specific mortality. However, due to the lack of population-wide surveillance data on the rate of recurrence, CRC-specific mortality is the most often used outcome to measure population-wide progress in CRC treatment. The use of mortality as the primary surveillance endpoint may be suboptimal, as recurrent tumors are amenable to therapy, so not all patients who have a recurrence will die from it. Furthermore, focusing on mortality impedes the identification of factors related directly to recurrence. We have, for example, shown a different relation between use of statins and colorectal cancer recurrence versus colorectal cancer-specific mortality in the same cohort of early-stage colorectal cancer patients, suggesting that the two outcomes can yield different epidemiologic estimates of association.¹³ Direct study of recurrence is therefore often the preferable approach for clinical epidemiology studies of CRC outcomes.

Unfortunately, recurrence data are not routinely collected by most population-based registries, including the Danish registries. The Danish registry system does allow unambiguous linkage across registries,¹⁴ and these linkages facilitated the development of an algorithm to predict CRC recurrences using diagnostic and procedure codes across multiple registries.¹⁵ Development and use of such algorithms in epidemiologic research requires a validation substudy to compare quantitatively algorithm-defined recurrences with gold standard—clinically recorded—diagnoses. Validation substudies measure the accuracy of the algorithm’s classification of recurrences, and the validation parameters they produce can be used in quantitative bias analyses to bias-adjust the influence of measurement error on association estimates. However, methods for validation substudy design using prospectively collected data have not been fully developed. Historically, validation study design has received far less methodologic development than standard study designs, such as case-control or cohort studies. What limited research exists on validation studies often pertains to selecting sample sizes. Further, most previous guidance on designing validation substudies pertained to scenarios in which study enrolment and follow-up had been completed before the validation study has been implemented.^16–18 In this proof of concept study, we apply a recently developed validation study design for prospectively collected validation data to validate CRC recurrences identified through an algorithm against gold standard diagnoses from medical records in an actively followed cohort of CRC patients in Denmark. We use the adaptive validation design to model when sufficient validation data would have been collected to meet a predefined stopping rule and estimate the classification parameters.

Materials and Methods

Study Population

We included Danish CRC patients who underwent surgery and were registered with the Danish Colorectal Cancer Group (DCCG) database.¹⁹ All patients had been enrolled at Aalborg Hospital into an observation cohort or into a randomized trial. The observation cohort consisted of consenting patients admitted between October 2003 and November 2005 for intended curative surgery.²⁰ The COLOFOL randomized trial (clinicaltrials.gov identifier NCT00225641) enrolled in stage II and III patients from January 2006 to January 2011 and compared two follow-up regimens after surgical resection.²¹ Both cohorts received active biennial follow-up for colorectal cancer recurrence after completion of primary therapy, and this follow-up information was treated as the gold standard.

Analytic Variables

Recurrences: We defined CRC recurrences as tumor growth at or near the site of the original tumor and in the same organ, or metastases to tissue adjacent to the original tumor site, or metastases to a distant organ. The CRC recurrence prediction algorithm used four independent data sources.¹⁵ Briefly, recurrences based on the algorithm were defined as the presence of at least one of the following:

1. Danish National Patient Registry (DNPR)-registered or Danish Cancer Registry (DCR)-registered metastasis codes (ICD10 DC76–DC80), 180 days or more after first colorectal cancer surgery.^22,23
2. DNPR-registered cytostatic therapy codes, 180 days or more after first colorectal cancer surgery.
3. Danish Pathology Registry SNOMED code combinations recorded 180 days or more after first colorectal cancer surgery, and without a new primary cancer diagnosis registered in DNPR or DCR.²⁴
4. A code specific for local or regional colorectal cancer recurrence in the DNPR any time after primary diagnosis.

This algorithm was previously applied to a cohort of all Danish colorectal cancer patients to understand the population-wide descriptive epidemiology of colorectal cancer recurrence.⁷

Statistical Analysis

We used the actively followed CRC cohort’s recurrence diagnoses as the gold standard and performed two sets of validation studies to estimate positive and negative predictive values (PPV and NPV, respectively), and sensitivity and specificity classification parameters. Conditioning on the algorithm’s classification of recurrence status allowed calculation of the PPV and NPV. Conversely, conditioning on gold standard recurrence status allowed calculation of the sensitivity and specificity of recurrence classification. For each of the two validation sets, we used the framework of Bayesian monitoring methods to identify when a sufficient sample size had been obtained to meet a prespecified threshold value and level of precision for these four classification parameters.²⁵ Bayesian monitoring techniques have been used in clinical trials to estimate treatment efficacy while trial data accumulate, and to optimize randomization allocation schemes as data accrue. We extended this framework to prospectively collected validation data to optimize use of study resources when estimating classification parameters. Our main goal in this analysis was to model when sufficient validation data would have been collected to establish the validity of the algorithm, or, conversely, to determine whether to view the algorithm as untenable for use in classifying CRC recurrence. Although the data from the actively followed CRC cohort were already complete, we simulated the two validation scenarios as if the data were being collected prospectively. To estimate the PPV and NPV, validation study members were ordered chronologically by the date of their algorithm-ascertained CRC recurrence. Similarly, to estimate the sensitivity and specificity, validation study members were ordered chronologically by the date of their CRC recurrence recorded from active follow-up. This ordering allowed for the selection of participants and calculation of classification parameters as if validation data were accruing simultaneously with other cohort data over follow-up.

Once cohort members were chronologically ordered, we used an iterative beta-binomial Bayesian model, which updates the PPV/NPV or sensitivity/specificity, which we designate θ, over regular follow-up intervals. Each classification parameter is a proportion, so can be modelled with a beta prior distribution and binomial likelihood. We began by assigning a noninformative beta prior for each parameter, , where is the validation parameter of interest. We update this distribution iteratively with the likelihood contributed by new validation data (y_j) to calculate a new posterior distribution for each parameter for each sample:

Each time interval represents the updated classification parameter after new validation data have been added to the prior validation data collected, and this iterative process can be continued until stopping criteria have been met or until data collection is complete. To establish intervals after which to update estimates, we evaluated the design under two scenarios. First, once five algorithm-ascertained recurrences had accumulated over the follow-up period, we selected those five individuals and also, at random and without replacement, we sampled five individuals without algorithm-ascertained recurrences from the set of cohort participants without an algorithm-ascertained recurrence by that point in the study’s follow-up (i.e., a 10-person validation set at each time interval). These were used to estimate the PPVs and NPVs, updating the estimate from the preceding time interval. This updating was done iteratively over the complete study period. We used identification of five recurrences to trigger the computation because recurrences were (and were expected to be) much less common than lack of recurrence, so the latter were always relatively abundant. Second, we repeated the analysis using 10 recurrences instead of five to trigger the computation (ie, a 20-person validation set at each time interval). Use of five or ten recurrences to trigger computation was meant to simulate the accrual of sufficient sample (10 or 20) to justify sending a medical record abstractor to a site to complete the validation. In the second validation set, we repeated these two scenarios using the recurrences recorded from the actively followed cohort, which allowed for estimation of sensitivity and specificity.

In each validation substudy, we assumed that sufficient validation data had been collected once the classification parameters met 1) a predefined threshold and 2) a corresponding level of precision. When estimating the PPV and NPV, we assumed that the validation sample was sufficiently large if the lower credible bound (2.5^th percentile) of both the PPV and NPV exceeded 0.80. We likewise assumed that validation was futile if the upper credible bound (97.5^th percentile) was less than 0.80. The choice of 0.80 as the threshold was informed by the content. Assuming that the algorithm would be considered valid if the point estimate for PPV and NPV were near 0.90, then the lower bound for the CI would need to reach at least 0.80. For the precision criterion for PPV and NPV, we assumed that validation efforts would cease if the credible interval width—defined as the absolute difference between the upper and lower CI—was less than 0.15. In our estimation of sensitivity and specificity, we used results from the Danish population-wide cohort study to inform an allowable false-positive rate.⁷ In this population-based study, the observed 5-year risk of recurrence, using recurrences identified from the algorithm, was approximately 25%. If we assumed that the true risk of recurrence has a lower bound of 20% (informed by content), this limits the false-positive rate to 0.10, leading to the cut-off of ≥0.90 for the 2.5^th percentile of the lower credible bound for both the sensitivity and specificity classification parameters as an indicator that validation had been optimized. For the precision criterion, we assumed that validation efforts could cease if the credible interval width was less than 0.08.

Results

The combined CRC cohort and clinical trial cohorts used for this analysis included 355 CRC patients, of whom 63 (18%) had a recurrence detected by gold-standard active follow-up. According to the predictive algorithm we are validating, there were 70 (20%) recurrences over the course of follow-up. In the complete validation data,¹⁵ the PPV and NPV at the end of the study period were 0.86 (95% CI 0.75, 0.93) and 0.99 (95% CI 0.97, 1.00), respectively (Table 1). The sensitivity and specificity of classification of recurrence at the end of follow-up were 0.95 (95% CI 0.87, 0.99) and 0.97 (95% CI 0.94, 0.98), respectively (Table 1).

Table 1 Estimates of PPV, NPV, Sensitivity and Specificity Classification Parameters for Possible Outcome Misclassification of CRC Recurrence from the Adaptive Validation Substudy Design

PPV and NPV Classification Parameters

When recurrences were identified by the algorithm, the adaptive validation design identified when sufficient information had been collected to cease validation efforts (Table 1 and Figure 1). When 10 individuals were included in the validation set at each time interval, the adaptive validation approach reached the stopping rule for threshold and for precision after validation data from 110 individuals had been collected. The final estimated PPV and NPV classification parameters were 0.89 (95% CI 0.80, 0.95) and 0.96 (95% CI 0.90, 0.99) respectively, which are good approximations to the values obtained from the complete data (Table 1). When 20 individuals were included in the validation at each time interval, the adaptive validation approach identified that the stopping criteria had been reached after validation information from 120 persons had been collected. The final estimated PPV and NPV classification parameters were 0.90 (95% CI 0.82, 0.96) and 0.98 (95% CI 0.94, 1.00), respectively, which are again good approximations to the values obtained from the complete data (Table 1). In general, estimates were more precise when 20 individuals were sampled at each update step.

Sensitivity and Specificity Classification Parameters

Similar to the PPV and NPV validation, the adaptive validation method identified when sufficient validation efforts had been expended, although this did not reach the desired stopping criteria when 10 individuals were included in the validation set at each time interval. When 10 individuals were included in the validation set at each time interval, the estimates of sensitivity and specificity were closest to the criteria for stopping once 120 individuals had their recurrence information validated. The sensitivity and specificity estimates were 0.95 (95% CI 0.89, 0.99) and 0.95 (95% CI 0.89, 0.99), respectively (Figure 2), which are good approximations to the values obtained from the complete data. When 20 individuals were included in the validation set at each time interval, the estimates of sensitivity and specificity met the criteria for stopping once 120 individuals were included, with improved precision over the 10-person validation approach. The sensitivity and specificity estimates were 0.95 (95% CI 0.89, 1.00) and 0.98 (95% CI 0.94, 1.00), respectively (Figure 2), which are again good approximations to the values obtained from the complete data.

Discussion

In this proof of concept validation study, we illustrate that the adaptive validation approach readily optimized the estimates of classification parameters for the ascertainment of CRC recurrences by a predictive algorithm. This evaluation was carried out for the PPV, NPV, sensitivity, and specificity—all of which showed that validation information from only ~120 individuals was needed to meet the predefined threshold and precision criteria of the estimates of the parameters of interest. As the original development of the algorithm used the actively followed cohort (n=355) as a validation substudy, we knew the overall classification parameters, allowing comparison of the parameters estimated using the adaptive design to the underlying parameters when the entire cohort’s complete data had been validated. These results suggest that 235 of the 355 cohort members (66%) added only marginal value to the validation study results. Were these 235 validated by a resource-intensive strategy, such as medical record review, use of an adaptive validation design would have saved considerable research resources.

As interest expands in cancer recurrences as an endpoint for cancer survivorship studies, population-based registries without standardized collection of recurrence information may apply similar algorithms with multiple data streams to identify recurrences.^26,27 Such an undertaking requires extensive and often expensive validation, and monitoring of the validation results as well the expenses allocated to them. Previous methods to design validation studies, such as the simple random sample or balanced design, require completed data collection before implementation.¹⁷ The adaptive validation design allows thoughtful consideration of which validation data will be most informative for the study as follow-up data accrue. Through iterative updating of estimates of bias parameters, researchers can ensure a purposeful design of internal validation substudies, stopping validation data collection when additional information will add only marginal value or when additional information is likely to be futile. This approach may save researchers valuable resources that can be redirected to additional data collection without sacrificing the quality of the validation data collected. For prospectively collected data, this design allows the simultaneous implementation of the validation with the collection of the cohort’s data instead of the traditional study designs that require waiting until all data have been collected before beginning validation.

This study is not without limitations. We used data that had already been collected and validated, then simulated the validation studies as if they were happening in real time, so estimates of the classification parameters from the complete data were known. The advantage was that use of a complete validation study allowed a comparison of estimates obtained when the stopping rule was met with estimates in the complete data. However, our choice of stopping point was inevitably informed by knowledge of the results from the complete data, which would not be available when undertaking a truly prospective design. In that case, the stopping rule would have to be informed by another strategy or by content-specific knowledge, possibly from similar validation studies in external populations or desired endpoints. Additionally, validation data were only available from two actively followed patient populations at one hospital site in Denmark, which may have led to bias in the classification parameters if there are regional differences in clinical practice. Furthermore, we did not base our stopping rule on an estimate of an exposure-outcome association of interest, which would have changed our sampling strategy by requiring sampling and estimation of at least the predictive values within categories of the exposure. An interesting finding was that the majority of the misclassified individuals occurred in the last time period of the study’s follow-up, which may suggest that the actively followed cohort lost patients to follow-up before a recurrence could be documented, and that the accuracy of the algorithm was better than previously reported (Supplemental Figures 1 and 2).¹³ The previous validation compared the cumulation of recurrences in the actively followed cohort using the complete data at the end without insight into the trend in classification over time. Our results indicate that the trend over time may be an important consideration, which was only apparent when we evaluated the time-dependence of the estimates of the classification parameters.

Conclusions

In this proof of concept application of the adaptive validation study design for outcome misclassification, we demonstrated the ability of the method to accurately determine when sufficient validation data have been collected. This method serves as a novel validation substudy design for prospectively collected data with simultaneous implementation of a validation study.

Figure 1 Adaptive validation using (A) 10-person and (B) 20-person validation at a time among CRC recurrences identified through the algorithm to estimate the positive and negative predictive values.

Figure 2 Adaptive validation using (A) 10-person and (B) 20-person validation at a time among CRC recurrences identified in the actively followed cohort to estimate the sensitivity and specificity.