Assessing the Accuracy of Symptoms and Adverse Events Reporting for Lung Cancer Treatment in the Danish National Patient Registry

Background

Lung cancer is one of the most frequently diagnosed cancers worldwide.¹ In Denmark between 2017 and 2022, an average of 71.3 per 100,000 people were each year diagnosed with lung cancer and an average of 49.9 per 100,000 deaths per year due to lung cancer were reported.²

Currently available treatments for lung cancer include systemic anti-cancer therapies (SACTs), surgery, radiation therapy (RT), or a combination of these.^3,4 In Denmark, the treatment strategy depends on the type of lung cancer, stage, and patient performance status.⁵ Adverse events of SACTs depend on the treatment modality, whereas those of RT depend on the treatment site. Commonly observed adverse events of SACTs include fatigue, nausea, vomiting, hair loss, fever, and nerve damage, whereas the most common adverse events of RT are fatigue and symptoms of tissue irritation from the irradiated area.^6,7 These adverse events are detrimental to the quality of life of patients and can even be counterproductive if the patient does not survive long enough to benefit from the treatment.

To assist clinicians in the decision-making processes, the development of decision support tools for predicting the short-term mortality of patients with lung cancer has been investigated.⁸ This decision support tool leverages extensive medical data to provide data-driven predictions including symptoms and adverse events. Data on symptoms and adverse events were obtained from the Danish National Patient Registry (DNPR).⁹ DNPR contains hospital administrative data, including diagnoses and treatments, and is widely used in epidemiological studies. Its validity has been demonstrated for SACTs,¹⁰ but it is unknown how much information concerning the symptoms and adverse events of anti-cancer treatments is available, notably for lung cancer.

In this study, we assessed the validity of reporting symptoms and adverse events in the DNPR to evaluate the usability of these data in decision support tools but also more broadly for epidemiological studies.

Methods and Materials

Patient Selection

The initial patient cohort consisted of 2380 lung cancer patients in contact with the Department of Oncology, Aalborg University Hospital, between 1991 and 2022, and with a documented date of death between 2015 and 2022. This ensures at least 24 years of potential history for each patient, hereby minimizing potential survival bias. Of the 2380 patients, 450 were excluded because they did not receive any form of anti-cancer treatment, defined as chemotherapy, radiotherapy, or surgery, leaving 1930 patients. These 1930 patients are referred to as the original cohort in the following sections.

Ninety-nine patients were randomly selected from the original cohort for further analyses of the patient journals.

Upon further inspection, four patients were excluded due to anti-cancer treatment for different cancer types. Consequently, this study was based on patient journals of 95 patients, referred to as the study population.

Pseudonymization was performed by replacing each patient’s civil registration number (CPR number) with an incremental patient ID. In addition, names were removed from the text.⁹

Data Extraction

Patient Journals

Data on lung cancer patients were extracted from pseudonymized patient journals by four raters. The raters were undergraduate students in Medicine and Medicine with Industrial Specialization at Aalborg University. The rating was performed under the supervision of an experienced oncologist at Aalborg University Hospital. The extracted data, referred to as journal data, consisted of patient ID, predefined category, the exact wording of how the category was described, grading, and date of occurrence. The exact wording was used to assign the entry to one of the eleven categories of symptoms and adverse events, hereafter referred to as clinical manifestations, which were selected based on their relevance and frequency in a randomly selected subset of 12 patient journals from the study population.

Twelve journals were selected to ensure equal distribution among the four raters and to maintain a sample size above ten journals, which was considered sufficient to identify common clinical manifestations in the patient journals. The 11 clinical manifestations included constipation, coughing, diarrhea, dyspnea, fatigue, hemoptysis, headache, hoarseness, nausea, pain, and weight loss.

The grading was performed according to international guidelines of the Common Terminology Criteria for Adverse Events (CTCAE)¹¹ with the purpose of standardizing the collection of data. The grading guidelines are listed in Supplementary Table 1. The data from 95 patients in the study population were extracted using this approach. Four patients from the study population were randomly chosen for review of their journal notes by all four raters. The remaining 91 journals were split as equally as possible among raters.

Patient Administrative System

The DNPR contains patient information notably diagnosis codes following the Danish Medical Classification System (SKS).¹² The codes used to identify clinical manifestations, as defined above, are listed in Supplementary Table 2. Data to populate the DNPR are provided by each Danish region. In the North Denmark Region (NDR), these data are from the Patient Administrative System (PAS) which was the data source used in this study.

As no grading was available from the DNPR data, only patient ID, clinical manifestations, and date were extracted. Thus, the registered symptoms and side effects were graded as > 0 and considered positive compared with the grading system of the extracted data.

Analysis

Interrater Agreement Analysis

Interrater agreement was evaluated between the four raters using Light’s kappa.¹³ To evaluate this, data entries from two raters at a time were matched based on the patient ID, clinical manifestation, and date. In cases where multiple assessments by the same rater for the same clinical manifestation were performed on the same day, only the assessment with the highest grading was retained. If a rater did not report any assessment on a day for a clinical manifestation, while other raters did, a value of −1 was used as grading for this rater. Confidence intervals for Light’s kappa were calculated by bootstrapping by resampling with replacement of the identified occurrences and their corresponding six possible dependent pairwise assessments 1000 times. Light’s kappa is an extension of Cohen’s kappa for scenarios involving more than two raters. Therefore, a standardized interpretation of kappa values was used to evaluate the results.¹⁴ Light’s kappa was interpreted as follows: Light’s kappa <0.00 indicates poor agreement, 0.00–0.20 slight agreement, 0.21–0.40 fair agreement, 0.41–0.60 moderate agreement, 0.61–0.80 substantial agreement, 0.81–1.00 almost perfect agreement, and 1.00 perfect agreement.^14,15 This interpretation allows for Light’s kappa > 0.41 to be considered acceptable.

Danish National Patient Registry Compared to Patient Journals

For entries manually extracted from the four patients shared across raters, a consensus among raters was used. This consensus was obtained by comparing journal ratings and grading of clinical manifestations based on the majority of equal gradings. The journals reviewed by only one rater were deemed accurate upon achieving an acceptable level of interrater agreement.

To evaluate the concordance between the DNPR and journal data, records from both datasets were generated, consisting of patient ID, clinical manifestation, and date. For each patient and clinical manifestation, at most one record per date was kept in both datasets, hereafter referred to as an occurrence.

Six time period aggregations were considered: aggregating on date (no aggregation), week, month, quarter, year, and patient (no temporal criteria) and keeping only one record per aggregated time period. Both datasets were subsequently matched for patient ID, clinical manifestations, and aggregated time periods. For example, if a patient had a DNPR entry for fatigue on Monday the 23^rd of March 2015, and fatigue was described in the patient journal on Tuesday the 31st of March 2015, this would count as a month, quarter, year, and patient match, but not a date and week match.

A matched DNPR record was considered a true positive, while a DNPR record that was not matched in the journal data was considered a false positive, and a non-matched journal data record was considered a false negative. In this setup, true negatives cannot be assessed.

Concordance was measured using the F1-score, which represents the harmonic mean between the sensitivity and the positive predictive value.¹⁶

Python 3.11 in Jupyter Notebook was used for data setup and analysis.

Approval

The Regional Council of the NDR approved this study under the Danish Health Act § 46, sec 2, under reference number 2022–044174. The study was registered in the NDR’s research project inventory as F2022-186 to ensure compliance with Article 30 of the Danish Data Protection Regulation.

Results

Study Population

The characteristics of the study population are summarized in Table 1. The original cohort and study population exhibited similar distribution. Both were slightly overrepresented by men, with most patients above 60 years of age at the time of diagnosis (87% of patients). The majority of the patients in all considered cohorts survived at least 1 year from first diagnosis. Regarding treatment, SACTs alone or in combination with other therapies were the most commonly observed (69% of patients).

Table 1 Study Population Characteristics

Interrater Agreement Analysis

Figure 1 shows the mean results of the interrater agreement analysis. The analysis compared the four raters’ assessments of clinical manifestations. The results showed that Light’s kappa for all clinical manifestations exceeded 0.65, except for constipation, which had a Light’s kappa value of 0.46. Notably, six clinical manifestations achieved Light’s kappa values > 0.80. Therefore, the agreement varies from moderate to perfect.

Figure 1 Interrater agreement analysis displayed by Light’s Kappa between all four raters for each clinical manifestation with 95% confidence intervals. Confidence intervals were obtained by bootstrapping and are displayed as blue bars around the red striped lines showing the mean Light’s kappa.

Frequency of Reporting

The frequencies of the clinical manifestations are listed in Table 2. Coughing, dyspnea, fatigue, pain, and weight loss were recorded in more than 90% of the patients, with pain being the most frequently reported clinical manifestation (1169 occurrences). Constipation, diarrhea, hemoptysis, headache and hoarseness are recorded in less than 70% of patients, with hoarseness being the least frequent reported clinical manifestation (83 occurrences). Although fatigue is the most prevalent clinical manifestation affecting all but one patient, it has only 585 occurrences, with an average of 6.2 occurrences per patient.

Table 2 Frequency of Clinical Manifestations

Danish National Patient Registry Compared to Patient Journals

Figure 2 shows a match of exclusively positive findings for the clinical manifestations. Overall, the precision is often high, whereas the recall tends to be low. The F1-score was low for nearly all clinical manifestations when aggregated by date, week, month, quarter, and year. When aggregating at the patient level, only dyspnea, hemoptysis, and pain had an F1-score above 0.25, with only dyspnea reaching above 0.5. This suggests a moderate level of reporting of these clinical manifestations at the patient level in the DNPR. Only dyspnea and pain reached an F1-score above 0.25 aggregating at the year level, and only dyspnea reached the threshold aggregating at the quarter level.

Figure 2 Count of occurrences in journal notes, the DNPR and matching between the two, and performance of precision, F1-score, and recall for pain, dyspnea, fatigue, coughing, weight loss, nausea, headache, diarrhea, constipation, hemoptysis, and hoarseness. Abbreviations: DNPR, Danish National Patient Registry.

F1-scores at the patient, year, and quarter levels are listed in Supplementary Tables 3, 4, and 5, respectively. Multiple clinical manifestations have an F1-score of 0, although specific SKS codes exist to report them. This indicates that most instances of clinical manifestations recorded by raters from patient journals did not appear in the DNPR.

Discussion

Main Results

This study compared data extracted from patient journals and the DNPR to evaluate how accurately the DNPR reflect clinical manifestations in lung cancer patients.

The high interrater agreement for most symptoms indicates that the manual journal data extraction was generally reliable, although the lower agreement for constipation (kappa = 0.46) suggest that some clinical manifestations are more prone to subjective interpretation.

The concordance between the journal data and the DNPR was consistently low across all time intervals, highlighting the limitations of the DNPR, as an administrative tool, for clinical manifestation reporting. At the patient level, only severe clinical manifestations, such as dyspnea, hemoptysis, and pain reached an F1-score >0.25, with only dyspnea reaching >0.5, suggesting a moderate level of reporting of these clinical manifestations at the patient level in the DNPR. This pattern that the DNPR primarily reflects clinical manifestations that are clinically significant and likely to influence treatment decisions or resource use, while minor or moderate clinical manifestations stay underreported.

Strength and Limitations

This population-based study analyzed a randomly selected population of patients with lung cancer at the Department of Oncology in the NDR. Random selection limits selection bias.¹⁷ However, including patients from only one region could lead to a lack of generalizability of the results to Denmark and the rest of Scandinavia.¹⁰

Additionally, the study population comprised 95 patients, representing a relatively small cohort, that could hinder the generalizability of the results. The size of the cohort, provided the scope of the project, was nevertheless deemed sufficient to provide an indication of the concordance between journal data and the DNPR.

We decided to include patients based, not on their diagnosis date, but on their death date. The rationale was to have a complete picture of each patient trajectory, from diagnosis to death. This design choice could be seen as conditioning on future events but death being an ineluctable outcome for all patients, we considered it as acceptable since there was a limited patient selection on diagnosis date. Indeed, thanks to the long available history, having access to all patient data from 1991, we ensured a limited selection bias based on survival, allowing us to include patients with diverse survival profiles, as illustrated in Table 1. Otherwise, it could have led to an overrepresentation of the most severe cases, reducing the generalizability of the findings.

The choice to scrutinize 12 patient journals to identify clinical manifestations can be considered arbitrary. Indeed, this was not based on a proper power calculation before the study, as it was intended to be exploratory. The assessments from these journals could possibly be biased owing to more thorough examinations compared with the remaining patients.

A limitation of this study could also be seen in the reliability of the reference standard. In this study, manually extracted patient journal data were considered the ground truth, leading to methodological challenges. First, the raters professional background may have influenced data quality. Although supervised by an oncologist, students may be more prone to misinterpret clinical notes and the severity of the clinical manifestations, which could be the cause of the low interrater agreement for constipation (Light’s kappa = 0.46). Second, the clinical manifestations reported by the patients were selectively recorded by clinicians, depending on their clinical relevance and significance. Thus, the patient journals themselves can be inherently incomplete and non-standardized leading to documentation bias. Third, the two data sources use different systems for describing clinical manifestations. The gradings based of CTCAE shows how severe the clinical manifestation is, while the DNPR codes only tell us if the clinical manifestation is present or not. In this study, any CTCAE grade above 0 was treated as equal to a DNPR entry, but this may be seen as an oversimplification.

Interrater Agreement

Interrater agreement analysis was conducted using Light’s kappa to assess the similarity of ratings by the raters across 11 clinical manifestations in the patient journals of the four patients.

The interrater agreement varies among clinical manifestations, as shown in Figure 1. This variation could be attributed to several factors. First, some clinical manifestations are more difficult to grade because of an insufficient level of detail in patient journals. Second, the total number of clinical manifestations extracted varied among raters. Human error, either by overlooking a clinical manifestation or documenting one that is not present, may account for missing or added value. Therefore, Light’s kappa was calculated without accounting for the different numbers of clinical manifestations noted by each rater.

When employing Light’s kappa, an extension of Cohen’s kappa, it is expected that the data will be on a nominal scale.¹⁸ However, only a few gradings adhered to this scale, while the majority were on an ordinal scale. In addition, the standardized interpretation of kappa values is criticized when used in health-related studies because it allows for a kappa of 0.41 to be acceptable.¹⁴ Therefore, it is suggested that a kappa value ≥0.60 is the acceptable standard in health-related studies.¹⁴ Thus, the only clinical manifestation with an unacceptable agreement was constipation (Light’s kappa = 0.46). These findings indicate that the manually extracted data from patient journals are reliable. However, as the mean Light’s kappa value for constipation is under the acceptable standard in health-related studies, manually extracted data on this cannot necessarily be considered the truth.

Concordance Between Patient Journals and the Danish National Patient Registry

This study explored whether the DNPR could serve as a source for clinical manifestations for decision support tools and future research. While the DNPR did not reflect the full spectrum of clinical manifestations documented in patient journals, it did reflect, in some cases, severe and clinically significant manifestations, such as dyspnea, hemoptysis, and pain. Dyspnea and pain were the most frequently reported and showed the highest concordance with journal data. These findings align with a study by Chowienczyk et al who identified dyspnea and cough as common presenting symptoms in lung cancer patients from 2000 to 2017.¹⁹ Although coughing was documented in most patient journals, its low F1-score indicates that it is less consistently recorded in the DNPR. In addition, Chowienczyk et al found that the presence of chest pain is dynamic and decreases over time.¹⁹ However, the current study generally considered pain beyond the chest pain.

It is important to interpret these findings in the light if the DNPR’s primary purpose. As a primarily administrative tool, its coding practices prioritize diagnoses, treatments, and clinically significant events that influence health care resource allocation, rather than providing the detailed symptom documentation typical of clinical trial case forms.²⁰

Clinical Manifestations Report in Decision Support Systems

Clinical manifestations are crucial variables for assessing the quality of care and potentially predicting outcomes. While technically possible, the reporting of these data in the DNPR is only sparse, making them challenging to integrate in predictive models. Journal notes contain much more detailed information that could be used as input from predictive models leveraging Natural Language Processing (NLP) methods, notably to extract clinically relevant information.

Conclusion

This study found limited concordance between manually extracted data from patient journal notes and data from the DNPR. This highlights the limitations of using the DNPR to obtain reliable data for symptoms and side effects for lung cancer patients. Moderate concordance was nevertheless observed for dyspnea, hemoptysis, and pain at the patient level. This study points toward the potential for NLP to be applied for the extraction of clinical manifestations directly from journal notes to obtain more detailed data for epidemiological studies or to build machine learning-based decision support tools.