Validation of a Diagnostic Model to Differentiate Multiple Myeloma from Bone Metastasis

Plain Language Summary

It is often challenging to distinguish between multiple myeloma (MM) and bone metastasis (BM) due to their shared characteristics. Consequently, misdiagnoses and inappropriate referrals are common. To address this issue, a diagnostic model was recently developed to differentiate between these two conditions. In this multi-center validation study, we conducted the first validation of the MM-BM DDx model. The model exhibited preserved performance and robust accuracy, supporting its utilization in clinical practice. By incorporating the MM-BM DDx model, clinicians can promote appropriate and timely referrals of patients with a high likelihood of MM to hematologists. This approach has the potential to reduce the duration of MM diagnosis and treatment, leading to improved prognosis and survival outcomes.

Introduction

Multiple myeloma (MM) and secondary bone metastasis (BM) are two primary differential diagnoses for adult patients presenting with bone pain and destructive bone lesions.¹ An accurate initial diagnosis is crucial, as these conditions have major distinctions in terms of disease prognosis, subsequent investigations, and therapeutic goals.² However, in practice, it is often challenging to differentiate between these two conditions due to their similar clinical and radiographic characteristics.^3–5 Consequently, misdiagnosis at the first medical contact and inappropriate referrals are common.^6,7 Such occurrences have a significant impact, particularly on MM patients, as delayed myeloma diagnosis has been associated with substantial morbidity and worsened survival.⁶ Considering the continuous rise in incidence rates of both MM and BM in recent years,^8,9 the number of patients presenting with destructive bone lesions is expected to increase. This poses a significant challenge for primary care physicians who are typically the first to encounter such patients. Therefore, there is a need for a clinical diagnostic tool that can estimate the likelihood of MM and guide physicians in determining whether referral to hematologists is necessary.⁷

In 2020, a diagnostic tool to differentiate MM from BM in patients with destructive bone lesions (MM-BM DDx) was developed in a large series of patients diagnosed with MM and BM in northern Thailand.^10,11 The model includes three simple, routinely available laboratory parameters: serum creatinine, serum globulin, and serum alkaline phosphatase (ALP). The model demonstrated excellent discriminative ability and calibration. However, its reproducibility and applicability outside the original dataset were still unknown. Therefore, the objective of this study was to externally validate the performance of the MM-BM DDx model using a multi-center validation dataset that included patients from six tertiary care centers across Thailand.

Materials and Methods

Source of Data and Participants

This external validation study is a multi-center, retrospective cross-sectional study. Six university-affiliated hospitals and medical schools participated in this study: Faculty of Medicine, Chiang Mai University, Khon Kaen University, Siriraj Hospital at Mahidol University, Queen Savang Vadhana Memorial Hospital, Khon Kaen Hospital, and Chulabhorn Hospital. The development dataset of the MM-BM DDx model,¹⁰ including patients diagnosed with either MM or BM at Chiang Mai University Hospital from 2012 to 2015, was also utilized during this external validation to help interpret the findings.¹²

The Institutional Review Board of each center approved the study protocol (FAM-2564-07951) and waived the requirement of informed consent due to the retrospective nature of data collection. This study was conducted in accordance with the Declaration of Helsinki.¹³ We followed the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) statements for reporting.¹⁴

The intended patient domain for the MM-BM DDx model is an adult patient presenting with bone pain and destructive bone lesions on plain radiographs, whose differential diagnosis is presumably either MM or BM. The inclusion criteria were as follows: (1) patients older than 45 years old and (2) patients who were diagnosed with MM (International Classification of Diseases codes 10th revision (ICD10)-C90) or secondary BM (ICD10-C795). The decision to include only patients aged 45 years and older was based on the findings of the development study of the MM-BM DDx model.¹⁰ This study identified the specific age groups in which secondary BM and MM are the primary differential diagnoses for patients presenting with destructive bone lesions. Records with missing data were included. Each participating center retrieved the patient data between January 1, 2016, and December 31, 2022.

The MM-BM DDx Model

The MM-BM DDx model incorporates three clinical laboratory values: serum creatinine, serum globulin, and serum ALP. All three laboratory values undergo logarithmic transformation and are inputted into the MM-BM DDx fractional polynomial logit equation. The linear predictors (LP) of the model are summed to generate predicted log-odds, which are then converted to odds by exponentiation. The likelihood ratio (LR) of being MM is estimated by dividing the predicted odds of MM by the odds of MM in the development dataset, which was 0.30. The detailed equation for predicting the odds of MM is provided in Table 1. The online web application for the MM-BM DDx model is accessible through the following link: https://tinyurl.com/mm-bm-ddx.

Table 1 Detailed MM-BM DDx Fractional Polynomial Logit Equation for Predicting Odds of Multiple Myeloma (MM)

The MM-BM DDx model does not incorporate radiographic pattern as one of its predictors due to the high potential for misclassification. Moreover, during the development phase, the inclusion of all patients, irrespective of their radiographic patterns, aimed to broaden the scope of the patient population and include those who were suspected by physicians to have either MM or BM, without relying on precise classification based on radiographic pattern.¹⁰ This approach was adopted to encourage the utilization of the MM-BM DDx model regardless of the radiographic pattern, taking into account that patients with MM can still exhibit osteoblastic or mixed patterns in their imaging findings.

Outcome

The diagnosis of MM and BM was based on the ICD10 codes (ICD10-C90 for MM and ICD10-C795 for BM). The investigators verified all retrieved records. MM diagnosis was verified according to the standard diagnostic criteria by the International Myeloma Working Group (IMWG).¹⁵

Data Acquisition and Definition

We used standardized, de-identified case record form for data collection. The data on patient demographics (ie, age and sex), bone radiographic patterns from the plain film (ie, osteolytic, osteoblastic, and mixed lytic-blastic lesion), and clinical laboratory values (ie, hematologic parameters (ie, hemoglobin, hematocrit), renal function test and electrolytes (ie, blood urea nitrogen, serum creatinine, serum calcium), liver function test (ie, serum protein profiles, and relevant enzyme levels)) were collected at the date of MM or BM diagnosis (or the date closest to the diagnostic date but before the date of definitive treatment). For patients with MM, we collected data on immunoelectrophoretic paraprotein type and serum β2 microglobulin for staging patients according to the International Staging System (ISS). The data on the primary cancer site were collected for patients with secondary BM. Finally, we extracted the patient characteristics and the performance indices of the MM-BM DDx from the development study¹⁰ for comparative purposes.

Statistical Analysis

Statistical analysis was performed using Stata 17 (StataCorp, USA). Assuming an expected discriminative ability at an area under the receiver operating characteristic curve (AUC) of 0.8 with an error margin of 0.1 and the MM:BM ratio of 1:3, a sample size of 1768 with at least 584 patients with MM was deemed sufficient according to calculation methods proposed by Riley et al¹⁶ Continuous data were presented as mean and standard deviation (SD) or median and interquartile range (IQR) according to their distribution. Categorical data were presented with frequency and percentage. Fisher’s exact test was used for the comparison of categorical data. An independent t-test and Mann–Whitney U-test were used to compare continuous data as appropriate. All P-values were two-sided and regarded as statistically significant at less than 0.05.

Records with missing data were included in the analysis to avoid selection bias and improve statistical efficiency. Multiple imputation with chain equation (MICE) and predictive mean matching (PMM) were used to impute missing predictor values.¹⁷ All predictors within the MM-BM DDx model together with the final diagnosis were included within the MICE model. Age, sex, hemoglobin level, radiographic patterns of the lesions, and centers were also included as auxiliary variables. We created 30 imputed datasets and combined their estimates using Rubin’s rules.¹⁸

We followed a three-step framework for external validation, recently proposed by Debray et al¹² First, the relatedness between the development and validation datasets was evaluated by estimating the discriminative ability of the membership model, which is a binary logistic model with the dataset (either development or validation) as a dependent variable and predictors of the MM-BM DDx model and MM diagnosis as independent variables. The discriminative ability of the model was expressed using AUC. We also estimated the LP of the MM-BM DDx model and its SD for both datasets, which were used to reflect the distribution of case mix in each dataset. Independent t-test and variance-comparison test were used to compare the mean and SD of the LP, respectively. Second, we evaluated the external model performance in three aspects: model discrimination, model calibration, and diagnostic accuracy. For each patient, we used the MM-BM DDx model to predict the odds of MM. The discriminative ability of the model’s predicted odds was quantified using AUC. For calibration, all predicted odds were divided into ten groups based on the reported cutoff values from the MM-BM development study.¹⁰ The predicted probability of MM was estimated for each group and plotted against the observed proportion of MM to create calibration plots. The expected:observed (E:O) ratio, calibration-in-the-large (CITL), and calibration slope were estimated. Subgroup analyses were conducted to examine the discriminative ability and model calibration by center. To examine the diagnostic accuracy, we estimated each group’s sensitivity, specificity, and LR of the MM diagnosis. Sensitivity and specificity were estimated using a binary logistic model, whereas LR was estimated by dividing the individual’s predicted odds by the odds of MM in the development dataset. Third, we interpreted the model validation results based on the first and second steps.

A post-hoc analysis was conducted to assess the performance of the MM-BM DDx model in distinguishing patients with MM from those with secondary BM originating from various primary cancers. Additionally, we planned to investigate the prognostic significance of the MM-BM DDx model for both MM and BM patients. This would involve examining the association between the log-likelihood ratio (logLR) of MM and the International Staging System (ISS), which is a standard prognostic staging system for MM patients,¹⁹ and the association between the logLR for MM and serum ALP tertiles for BM patients. One-way analysis of variance (ANOVA) was used to test the statistical difference of logLR across the three groups for both objectives.

Results

Participants

A total of 3018 patients diagnosed with MM or BM were included in the validation dataset (586 with MM and 2432 with BM). Overall, the mean age was 62.8 (SD 10.0), with a male-to-female ratio of 54:46 in the validation dataset. Compared with patients in the development dataset, the clinical characteristics of those in validation datasets were similar, except for the distribution of radiographic patterns and the availability of plain film (Table 2). The proportion of missing data for each characteristic and MM-BM DDx predictor are also shown in Table 2. In the validation datasets, there were 2378 (78.8%) patients with complete data on all of the MM-BM DDx model parameters. The distributions of the three parameters, categorized by diagnosis and datasets, are visualized using box plots (Figure S1).

Table 2 Clinical Characteristics of the Patients in the Validation Dataset and the Development Dataset

The majority of patients with MM had either ISS stage II (14.9%) or stage III (47.8%) (Table S1). The most common immunoelectrophoretic paraprotein subtype was IgG (46.8%), followed by IgA (15.5%), and light chain (11.1%) (Table S1). The three commonly reported primary sites of patients with secondary BM were the lung (28.3%), breast (16.5%), and prostate (15.8%) (Table S2).

Model Performance

Regarding the relatedness between datasets, the membership model showed an AUC of 0.56 with a 95% confidence interval (CI) between 0.53 and 0.58. Both datasets also had a similar mean of the LP (development −1.67 vs validation −1.82, p = 0.135). However, the SDs of the LP were significantly different (development 2.11 vs validation 2.50, p < 0.001).

For model discrimination, the MM-BM DDx model illustrated an AUC of 0.89 (95% CI 0.87, 0.90) in the validation dataset (Figure 1a), which was similar to that of development study at 0.90 (95% CI 0.86, 0.93). For model calibration, the average predicted probabilities of MM were similar to the observed proportions of MM (E:O ratio 1.05, CITL −0.103, and calibration slope 0.97). According to the calibration plot, the predicted probabilities and the observed proportions of MM increased concordantly with one another (Figure 1b). Subgroup analysis results by center for discriminative ability and calibration are presented in Figures S2 and S3.

Figure 1 The diagnostic performance of the MM-BM DDx model in the external validation dataset. (a) Model discriminative ability illustrated with area under the receiver operating characteristics curve (ROC). (b) Model calibration plots illustrating the agreement between the expected probability from the MM-BM DDx model and the observed probability of multiple myeloma in the external validation dataset.

Table 3 presents the sensitivity, specificity, and LR for each decile of odds in the validation and development dataset. The group-specific sensitivity, specificity, and LR in the validation dataset were similar to those of development study.

Table 3 Diagnostic Accuracy of the MM-BM DDx Model by Predicted Odds Deciles in External Validation and Development Dataset

In the post-hoc analysis, we focused only on the four most prevalent types of primary cancer sites for BM. The AUC values of the MM-BM DDx model for differentiating various cancers from MM were as follows: 0.90 (95% CI 0.88, 0.92) for lung cancer, 0.91 (95% CI 0.90, 0.93) for breast cancer, 0.87 (95% CI 0.85, 0.90) for prostate cancer, and 0.90 (95% CI 0.88, 0.92) for liver cancer. Among MM patients with available ISS data (n = 394), there was a significant association between the higher predicted logLR of MM and a higher ISS stage (P < 0.001) (Table S3 and Figure S4). In contrast, the lower logLR of MM showed a significant association with higher serum ALP tertiles in patients with BM (n = 1870) (P < 0.001) (Table S3 and Figure S5).

Discussion

Our study validated the MM-BM DDx model using an independent dataset comprising patients from six centers across Thailand. The MM-BM DDx model demonstrated a discriminative ability similar to that observed during the development phase. The predicted probabilities of MM also closely matched the observed proportion of MM, indicating that the MM-BM DDx model remained well calibrated even when applied to an external dataset. Additionally, when the predicted odds were categorized based on the reported ranges in the development study, the sensitivity, specificity, and LR values for each group were consistent with those reported in the development study.

The similarity in performance between the validation and development datasets can be attributed to their relatedness. According to the poor discriminative ability of the membership model (AUC 0.56), both datasets were strongly related and difficult to distinguish.¹² This was also reflected through the similarity in the mean LP and the outcome proportions between the two datasets. For this reason, it is unlikely that the model’s external calibration-in-the-large and calibration would be significantly affected.^12,20 As for the SD of the LP, datasets with larger SD should presumably carry higher discriminative ability than those with lower SD.^12,20 Although the SD of the LP in the validation dataset was significantly higher than that of the development dataset, the discriminative ability was almost equal. This could potentially be attributed to the influence of extreme globulin values observed in the validation dataset.

Overall, we recommend implementing the MM-BM DDx model in clinical practice, particularly in populations similar to our study base, such as Thais or Asian populations. The model should be used for patients presenting with differential diagnoses of either MM or secondary BM. In clinical practice, distinguishing between MM and BM can be challenging, even with advanced imaging techniques.^21,22 In such cases, the MM-BM DDx model can play a crucial role by providing attending physicians with an estimated likelihood of MM based on just three routinely available laboratory values. For clinical applicability, the MM-BM DDx model provided users with the predicted LR of MM instead of odds. The clinical suggestion of the MM-BM DDx model was based on the magnitude and direction of the patient’s predicted LR.²³ Patients with LR above 2.0 should be referred to hematologists for further evaluation. In contrast, patients with LR less than 0.5 may not require referral as secondary BM is more likely than MM. Patients whose LR was higher than 0.5 but less than 2.0 should be considered on a case-by-case basis, and referral to hematologists may not be necessary.

As two of the three predictors in the MM-BM DDx model, namely serum creatinine and serum globulin, are established prognostic factors for survival and treatment outcomes in MM patients,^24–28 the model predictions may also have predictive value for these outcomes. However, since follow-up data was not collected in this study, we were unable to directly examine the prognostic significance of our predictions. To indirectly assess the potential prognostic utility of the model, we investigated the association between the higher predicted log-likelihood ratio (logLR) of MM and a higher ISS. This association was straightforward, as patients with a high predicted logLR of MM typically have elevated serum creatinine and globulin levels at diagnosis, which are generally indicative of unfavorable survival outcomes. The MM-BM DDx model also incorporates serum ALP, which is a potential prognostic marker in BM patients. Studies have shown that higher serum ALP levels at BM diagnosis are associated with a higher preoperative metastatic tumor burden, increased osteolytic activity, and serve as a prognostic marker for one-year survival in both spinal and non-spinal metastatic diseases.^29,30 Given that patients with high serum ALP have a low logLR of MM in our model, it can be assumed that BM patients with very low levels of logLR of MM may have poorer survival compared to those with higher logLR of MM. It is important to note that these findings are indirect due to the absence of direct follow-up data in our study. Implications of the model and further longitudinal studies are required to provide direct answers.

There are some major strengths to our study. This study is the first to externally validate the MM-BM DDx model using a large multi-center cross-sectional data of patients who presumably presented with bone pain and destructive bone lesions diagnosed with MM and BM across Thailand. The total study size and the number of events were also achieved as planned and significantly larger than that of the development study. To date, the MM-BM DDx model remains the only multivariable clinical prediction model developed to differentiate adult patients with destructive bone lesions. It is essential to recognize the distinction in the predictive context compared to models or studies aiming to use laboratory panels or clinical predictors for MM screening in a primary care setting,^31,32 where patients are often asymptomatic or have not yet developed bone lesions. While there have been some studies conducted to address similar questions, those studies primarily relied on advanced imaging techniques.^4,5,21,22 In 2021, Annibali et al investigated the diagnostic value of serum ALP in differentiating between BM and MM in adult patients with bone lesions.³³ Their findings demonstrated that a low or normal value of serum ALP was highly predictive of MM, which aligns with the results from our validation dataset.

Limitations

First, all the data used were retrospectively collected and subjected to bias. However, as all of the predictors used in the MM-BM DDx model were objective laboratory values, the degree of bias should be non-trivial. Second, the data on the MM-BM DDx predictors were missing in 21% of the patients. To maximize statistical efficiency, we performed a MICE procedure to impute missing data assuming the data were missing at random. Third, some participating centers collected data on only a portion of available patients during the study period. Fourth, the cross-sectional study base was not population-analog or consecutive. Without problem-oriented medical records,³⁴ including the intended study domain who presented with bone pain and destructive bone lesions on imaging was difficult. Moreover, not all patients with MM would present with bone pain and destructive bone lesion, and some patients with BM may be incidentally diagnosed. Thus, we had to assume that all patients diagnosed with MM or BM shared similar clinical presentations, which we believed was reasonable as these conditions were common differential diagnoses for adult patients with destructive bone lesions.¹ In addition, most patients with BM were symptomatic,³⁵ and 50 to 70% of patients with MM presented with bone pain and destructive bone lesions.^36,37 Other studies exploring features to differentiate MM from BM also directly contrasted patients diagnosed with MM and BM.^21,33 Fifth, the included study populations were patients in tertiary care centers, which was inconsistent with the primary care patients’ intended domain. Finally, despite the robust performance of the MM-BM DDx model during this multi-center validation, its generalizability to non-Asian populations or individuals younger than 45 years old remains unknown. Furthermore, the potential clinical impact of its application in practice is yet to be determined. Further research should be conducted to investigate the clinical outcomes of patients after the model’s implementation.

Conclusion

The MM-BM DDx model has undergone its first external validation as a diagnostic tool for distinguishing MM from secondary BM in patients with destructive bone lesions. With its excellent discriminative ability, satisfactory calibration, and robust accuracy indices preserved, we strongly advocate for the utilization of the MM-BM DDx model in clinical practice. Its implementation can facilitate the appropriate and timely referral of patients with a high likelihood of MM to hematologists, potentially reducing the time taken for MM diagnosis and treatment and ultimately leading to improved prognosis and survival.