Use of Novel Open-Source Deep Learning Platform for Quantification of Ki-67 in Neuroendocrine Tumors – Analytical Validation

Introduction

Ki-67 is a widely used biomarker for quantifying cellular proliferation in tumors, including neuroendocrine tumors (NETs). The Ki-67 index is calculated by dividing the number of Ki-67 positive cells by the total number of cells evaluated in a tissue sample.¹ A higher Ki-67 index indicates a higher rate of cellular proliferation and is associated with a more aggressive tumor phenotype. In NETs, Ki-67 quantification is used to help determine the tumor grade and prognosis as well as to monitor response to treatment. However, it is important to note that Ki-67 is not specific for NETs and its expression can be influenced by other factors such as the stage of the cell cycle. Therefore, Ki-67 should be interpreted in the context of other clinical and pathological features.² Artificial intelligence (AI) can be used to support the detection and quantification of Ki-67 in NETs. This can be done using a variety of techniques such as deep learning algorithms, computer vision and image analysis. Deep learning is a type of AI that has shown promise in the diagnosis and prognostication of NETs.^2–4 Deep learning (also known as deep structured learning) is a subfield of machine learning based on artificial neural networks (ANNs) in which the statistic models are established from input training data.⁵ Deep learning algorithms can be trained on large amounts of medical imaging data to accurately identify and quantify Ki-67 positive cells in tissue samples using digital pathology images.³ The use of AI in Ki-67 detection has the potential to improve the accuracy and consistency of Ki-67 quantification, as well as reduce inter-observer variability. Additionally, AI can also reduce the time and resources required for manual Ki-67 quantification, allowing for a faster and more efficient evaluation of large numbers of tissue samples.⁴ Deep learning algorithms can also be used to predict patient outcomes and treatment response, by analyzing imaging and clinical data to identify predictive features associated with disease progression and survival. This can enable the development of personalized treatment plans for patients. However, it is important to note that deep learning algorithms in NET analysis are still in the stage of development and require validation. Also, comparison of AI-based Ki-67 quantification to traditional manual methods is needed to ensure its accuracy and reliability.^5–8

Various studies have been conducted in the past on Ki-67. These are all on whole slide images (WSIs).^1–4,6 We did this study on digital images made at 10x through microscope connected camera on open-source software. It is also the only AI immunohistochemical (IHC) scoring model available for free via cloud-native platform with user-friendly interface (https://deepliif.org) for anyone to upload their images and get results. We used this software on data of our own population and validated this open-source software. DeepLIIF model was not adjusted for our dataset. We used it out-of-the-box for this study. Original DeepLIIF was trained/validated on restained/co-registered scanned IHC and multiplex immunofluorescence-stained slides. DeepLIIF code, pretrained models, training/validation/testing datasets are publicly available via their GitHub (https://github.com/nadeemlab/DeepLIIF). We wanted to give a message that pathologist using microscope not having the facility of scanner can also get the benefits from AI based deep learning model. These are proof of concept studies mainly for research. No previous studies have been performed in low-resource settings with fully open-source state-of-the-art AI approaches that can generalize to low-quality microscope snapshots with large tumor coverage (10x). For comprehensive analytical validation, an expert pathologist (P1) manually counted the IHC+/- tumor cells in our neuroendocrine cancer images which is an extremely tedious/laborious task, especially when this is done at 10X resolution with much larger tumor coverage; no previous manual vs AI counting studies have been performed at 10X, most were limited to 40X with much lower cell count. Through this analytical validation, we wanted to showcase that the DeepLIIF model can easily be used for accurate estimate of Ki-67 index in much larger tumor coverage in low-resource settings, where scanners and commercial AI solutions are not accessible/affordable. This analytical validation gives us the confidence and sets up the stage for much large-scale clinical validation that we are undertaking now. Moreover, DeepLIIF provides an easy solution for developing region pathologists to take advantage of advanced AI solutions where they are needed the most with declining pathologist numbers and increasing patient load. This study was conducted to see the concordance between open-source AI-based DeepLIIF (https://deepliif.org) software Ki-67 quantification versus manual quantification in NETs.

Materials and Methods

The study was conducted at Liaquat National Hospital from 2nd June 2023 to 2nd August 2023 and included all previously diagnosed cases of NETs from Liaquat National Hospital Laboratory after taking ethical opinion from ethical review committee. Antibody stained for Ki-67 was used in the study. The digital images were prepared from microscope connected camera at 10x from the hotspot regions. A total of 100 images were prepared from hot spot areas. Five consultant histopathologists (P1 to P5) scored the 100 digital images using the eyeballing method. Out of five, one Histopathologist P1 who manually counted the cells and prepared the scores was considered as standard and this score was compared with the score of other pathologists including the AI software. Since the scoring for Ki-67 is only done in tumor cells, P1 counted only the Ki-67+/- tumor cells. As shown in previous studies,⁹ disagreement among pathologists on tumor cell annotations (due to clear large irregular cellular morphology) on IHC images is less than 20%, making single-pathologist counts as manual reference good enough for Ki-67 scoring purposes.

Different tools within the DeepLIIF user-interface were used to select tumor regions and exclude stromal regions (Figure 1). The software offers an exclusion/inclusion region-of-interest lasso/selection tool that was used to exclude all the stromal cells after running DeepLIIF, as shown in Figure 1. This was done manually by pathologist as these tools were provided by software. The values of manual and automated scores were compared for concordance. The results were prepared by the kappa method.

All statistical analyses were performed using SPSS software version 24. Normality of continuous data was assessed by using Kolmogorov–Smirnov test. The data was represented by Med [IQR]. Agreement of P1 (gold standard) with P2, P3, P4, P5 and AI was measured by applying Kappa analysis, and validity was assessed by correlation. ROC plotted and AUC evaluated for each variable to find the quality of test. Diagnostic accuracy was also calculated for P2, P3, P4, P5 and AI by using Ki-67 cutoff score as 20.

Results

By using Kolmogorov–Smirnov test, the distribution of P1, P2, P3, P4, P5 and AI were found to be non-normal (p value < 0.05). The descriptive statistics of these variables is reported in Table 1. In Table 2, the agreement measure of P1 was assessed with P2, P3, P4, P5 and AI to observe the inter-rater reliability. The kappa statistic was found as 0.798, 0.674, 0.589, 0.609 and 0.860 for P2, P3, P4, P5 and AI respectively which represents moderate strength of agreement with P4, good strength of agreement with P2, P3 and P5, and excellent strength of agreement with AI.

Table 1 Descriptive Statistics for P1, P2, P3, P4, P5 and AI

Table 2 Measurement of Agreement

Figure 2 shows that the finding of Ki-67 by P1 is strongly positively correlated with P2, P3 and P4 while very highly positively correlated with P5 and AI. Here Cronbach’s alpha = 0.981, which is indicating excellent internal consistency and Intraclass Correlation Coefficient (ICC) = 0.885 with 95% CI (0.848–0.916) which is indicating good reliability. In Figure 3, ROC curves were plotted for P2, P3, P4, P5 and AI to assess the quality of the test by taking P1 as gold standard. The quality of test was assessed by the value of area under the curve, evaluated by ROC, which indicated excellent test quality for all the pathologists and AI.

Figure 1 Digital image of Ki-67 neuroendocrine tumor prepared at 10x was uploaded in the software and the concordance between manual versus automated score was observed.

In Table 3, the diagnostic accuracy of P2, P3, P4, P5 and AI was assessed by taking P1 as gold standard and using Ki-67 cutoff score as 20. The diagnostic accuracy was calculated as 90%, 84%, 80%, 81% and 93% for P2, P3, P4, P5 and AI, respectively. The highest diagnostic accuracy was found for AI ie 93%, as compared to other pathologists.

Table 3 Sensitivity, Specificity, PPV, NPV and Diagnostic Accuracy of P2, P3, P4, P5 & AI Using P1 as Gold Standard

Discussion

NETs are a diverse group of neoplasms that originate from neuroendocrine cells dispersed throughout the body. NETs can occur in various organs, including the gastrointestinal tract (GIT), pancreas, lungs, and more, reflecting the widespread distribution of neuroendocrine cells in the human body.^10,11 These tumors exhibit a remarkable spectrum of clinical behaviors, ranging from indolent, slow-growing lesions to highly aggressive malignancies. Histopathological diagnosis plays a pivotal role in characterizing these tumors, enabling clinicians to determine their precise nature, grade, and potential for metastasis. One of the prognostic factors for GIT NETs is the proliferative grade of the tumor calculated using the proportion of tumor cells positive for Ki-67 immunostaining.^12–14

Traditional methods of quantifying Ki-67 expression in NETs rely on manual assessment by pathologists. This process can be time-consuming, subjective and prone to inter-observer variability.^14,15 Using a novel open-source deep learning platform for the quantification of Ki-67 in NETs has the potential to significantly enhance the accuracy, efficiency and reproducibility of the diagnostic procedure. The analytical validation is carried out to ensure that the results generated are reliable, consistent and comparable to established methods.^1,15 Neural networks and advanced machine learning techniques can be utilized to automatically analyze histopathological images and accurately identify and quantify the Ki-67 protein expression, which is a marker for cell proliferation and an important factor in determining the aggressiveness of the tumor.¹ The deep learning model is trained using the annotated dataset. The model learns to recognize patterns and features in the images that are indicative of Ki-67 expression levels. The model’s performance is then assessed using a separate unseen dataset. The results obtained from the deep learning platform should be compared to the gold standard, which involves manual assessment of Ki-67 expression by expert pathologists. This step is critical for establishing the accuracy and reliability of the deep learning platform.^16,17

Deep learning models can process large volumes of histopathological images quickly and accurately, potentially reducing the time required for assessment. This is particularly valuable in clinical settings where timely diagnosis is critical. Deep learning algorithms provide consistent and objective results, mitigating the inherent variability associated with manual evaluations. This leads to more reliable and reproducible quantification of Ki-67 expression.^18,19 Neural networks can detect subtle patterns in images that may not be easily discernible by human observers. This can lead to more precise quantification, especially in cases with varying staining intensities. Once trained, the deep learning model can be applied to a large number of images without diminishing the accuracy. This feature is particularly helpful when dealing with datasets that are too extensive for manual evaluation.^4,5,20 Accurate and reliable quantification of Ki-67 expression can provide clinicians with essential information to determine the tumor’s aggressiveness, prognosis and potential treatment strategies. Automation and standardization of this process can lead to faster diagnoses, improved treatment planning, and enhanced patient outcomes.^21,22

Recently, several studies demonstrated that deep learning algorithms could be used to generate automatic Ki-67 proliferation index with high sensitivity and specificity in breast carcinomas and pancreatic neuroendocrine neoplasms.^23–25 In this study, the median value was 25 for P1 and P2, 30 for P3, P4 and P5 and 21 for AI (Table 1). The Cronbach’s alpha was 0.981, which indicated excellent internal consistency and ICC was 0.885 with 95% CI (0.848–0.916) which indicated good reliability (Figure 1). The Interclass Correlation Coefficient Results for digital image analysis in another study were 0.706, 0.606 and 0.912 for one, two and three field of view respectively.³ A study by Swati et al showed that eyeball estimation method had the highest concordance rate (84.2%), followed by WSI analysis of the entire slide (73.7%).³

Figure 2 Validity by Correlation Matrix showing that the finding of Ki-67 by P1 is strongly positively correlated with P2, P3 and P4 while very highly positively correlated with P5 and AI.

Tang et al showed that the digital image analysis (DIA) and manual count were highly concordant (ICC = 0.98). The ICC between DIA and the mean eyeball estimation of all observers was 0.88. The ICC for intra-observer consistency was 0.39±0.26.²⁰

In another study, the digital image analysis Ki-67 showed excellent agreement with the Ki-67 index in the routine pathology reports with 95% confidence interval (CI): 0.94–0.96. The observed kappa value was 0.86 (95% CI: 0.81–0.91).²¹ In our study, the strength of agreement was good for P2, P3 and P5, moderate for P4 and excellent for AI. The kappa values observed in this study were 0.798, 0.674, 0.589, 0.609 and 0.860 for P2, P3, P4, P5 and AI, respectively (Table 2). Sarag et al conducted a study in which digital image analysis of Ki-67 showed excellent correlation with manual counting.²⁶

The diagnostic accuracy in our study was calculated as 90%, 84%, 80%, 81% and 93% for P2, P3, P4, P5 and AI, respectively. The highest diagnostic accuracy was found for AI ie 93%, as compared to other pathologists. The sensitivity was 88.89%, 77.78%, 68.89%, 68.89% and 97.78% for P2, P3, P4, P5 and AI, respectively. The specificity was 90.91%, 89.09%, 89.09%, 90.91% and 89.09% for P2, P3, P4, P5 and AI, respectively (Table 3 and Figure 2).

Figure 3 ROC Plots prepared for pathologists P2, P3, P4, P5 and AI to access the quality of test by taking Pathologist P1 score as gold standard.

Therefore, developing an automated counting method using digital image analysis can enhance the scoring process and aid the pathologist, but requires careful modification of the software as it can induce errors such as counting stromal cells or lymphocytes and show discordant results due to background nonspecific staining.^27,28

We have done studies in past on simple digital images on various pathologies.^10,29–31 In doing so, we unleash the power of AI on simple digital images in resource limited setup. The results of above-mentioned studies were highly encouraging and tried to give a way forward to pathologists working in developing countries to adopt digital and computational pathology in resource limited setup.¹⁰

We wanted to give a message that pathologist using microscope not having the facility of scanner can also get the benefits from AI-based deep learning model. These are proof of concept studies mainly for research. No previous studies have been performed in low-resource settings with fully open-source state-of-the-art AI approaches that can generalize to low-quality microscope snapshots with large tumor coverage (10x). For comprehensive analytical validation, an expert pathologist (P1) manually counted the IHC+/- tumor cells in our neuroendocrine cancer images which is an extremely tedious/laborious task, especially when this is done at 10x resolution with much larger tumor coverage; no previous manual vs AI counting studies have been performed at 10x, most were limited to 40x with much lower cell count. Through this analytical validation, we wanted to showcase that the DeepLIIF model can easily be used for accurate estimate of Ki-67 index in much larger tumor coverage in low-resource settings, where scanners and commercial AI solutions are not accessible/affordable.^32,33 This analytical validation gives us the confidence and sets up the stage for much large-scale clinical validation that we are undertaking now. Moreover, DeepLIIF provides an easy solution for developing region pathologists to take advantage of advanced AI solutions where they are needed the most with declining pathologist numbers and increasing patient load.

Conclusion

The integration of a novel open-source deep learning platform for the quantification of Ki-67 in NETs has the potential to revolutionize the diagnostic process and the medical community can confidently embrace this technology as a valuable tool in the assessment and management of these tumors. However, validation of an open-access AI platform for immunohistochemistry is crucial to ensure that the tool provides accurate and reliable results.