Derivation of a Risk Assessment Tool for Prediction of Long-Term Pain Intensity Reduction After Physical Therapy

Introduction

In the United States, musculoskeletal pain is increasingly prevalent and is a leading contributor to disability.¹ The causes of musculoskeletal pain and its continued persistence are multifactorial,² with sociodemographic, psychological, and health-related factors being known contributors.³ Given the multidimensional nature of pain, predicting outcomes is difficult but integral to improving clinical decision-making at the point of care. This is especially important in emerging value-based care models that emphasize minimizing the risks of unwarranted care escalation, high costs, and poor outcomes.

Preliminary research on the use of predictive modeling techniques to enable healthcare providers to make more data-driven clinical decisions has been promising.^4–7 But integrating predictive modeling into the clinical workflow remains a challenge due to difficulties in translating predictive model results efficiently and meaningfully for the clinician and patient. The gold standard for implementation of predictive models in clinical care is through the use of risk assessment tools embedded in the electronic health record and available for use within the clinical workflow. Risk assessment tools quantify the risk or benefit of treatment using information known to predict clinically relevant outcomes. These tools are ideally informed by predictive models with variables that are routinely collected.

Existing risk assessment tools that take into account the multidimensionality of persistent musculoskeletal pain tend to incorporate pain-related psychological distress.^6,8 This is because psychological factors have a strong influence on treatment outcomes, particularly the development of persistent pain. The Optimal Screening for Prediction of Referral and Outcome Yellow Flag (OSPRO-YF) screening tool⁹ was developed with this purpose in mind. It concisely and accurately estimates how patients will score on 11 different full-length psychological questionnaires that represent a broad range of pain-related vulnerability and resilience constructs. Emerging evidence also suggests that prediction of pain-related outcomes may be enhanced by considering change in health and mental status such as psychological distress over time. Such an approach can improve prediction accuracy but must be weighed against the additional administrative burden of serial assessment of measures. Existing outcome prediction or risk assessment tools predicting persistent musculoskeletal pain do not incorporate change in psychological distress and dynamic nature of risk assessment over time.⁸ This limits the potential of risk assessment tools to quantify how risk changes as a result of treatment, not just risk assessment at baseline.

Risk assessment tools encourage probabilistic thinking, which is well suited for prognosis decisions, compared to deterministic thinking, which is better suited for diagnostic decisions. Within the context of musculoskeletal pain, one key outcome is whether pain reduction occurs after an episode of physical therapy care (ie, multiple treatment sessions). A risk assessment tool that can concisely and accurately communicate the probability of pain reduction while carefully weighing model accuracy against a manageable number of predictors is desired to increase the likelihood of clinical uptake. Therefore, the purpose of this analysis was to develop a multidimensional risk assessment tool—the Personalized Pain Prediction (P3) risk assessment tool—to quantify the risk of not achieving a 50% reduction in pain intensity 12 months after an episode of physical therapy. We selected a 12 month reduction because it would allow for prediction of risk for longer term outcomes that might be reflective of longer term treatment effects. The P3 tool is intended for physical therapists to improve decision-making for patients presenting with musculoskeletal complaints. Existing predictive approaches have estimated the risk of persistent disability from low back pain (STarT Back Screening Tool),⁸ chronic low back pain (PICKUP tool),⁶ persistent musculoskeletal pain¹⁰ (Beneciuk et al) and days to recovery from acute low back pain,¹¹ but have not included pain intensity reduction (ie, change in pain) as an endpoint. We speculate the prior focus of prediction tools on disability and persistence of pain is because these endpoints are of importance to the patient and often are used as primary endpoints in clinical trials. Similarly, pain intensity reduction is an endpoint that is important to the patient and is also used as a primary endpoint in clinical trials. Therefore, we chose a 50% reduction in pain intensity as our primary outcome to reflect a meaningful benchmark to represent a favorable treatment response.¹² We had the following primary aims: (1) to determine the predictive accuracy of two different approaches to model development—a longitudinal predictor approach that included modeling a set of predictors to optimize predictive accuracy that includes all possible OSPRO follow-up scores, and a reduced follow-up predictor approach that included a modeling of pain-related predictors and short-term OSPRO follow-up score that is most consistent with timing of care episodes—and (2) to compare the accuracy and clinical feasibility of these models to determine which model would best inform the P3 tool.

Methods

This was a secondary analysis from the Optimal Screening for Prediction of Referral and Outcome (OSPRO) validation cohort, which has been previously described.¹³ The purpose of the current analysis is to select a risk prediction model for 50% reduction in pain intensity, with the purpose of incorporating this model into risk assessment tool and associated interface to be used by clinicians to aid in clinical decision-making. This study was approved by Duke University institutional review board (Pro00104774). All participants provided written informed consent to participate. This study was conducted in compliance with the Declaration of Helsinki.

Participants and Eligibility

Participants seeking physical therapy treatment for a primary complaint of neck, low back, knee, or shoulder pain were recruited from the Orthopaedic Physical Therapy Investigators Network, a nationwide network of clinics participating in the OSPRO cohort study. Specific eligibility criteria have been detailed elsewhere¹⁴ and were set to be broadly inclusive of patients seeking physical therapy for common musculoskeletal pain complaints. In this cohort study, all data were collected electronically directly from patients via a Research Electronic Data Capture (REDCap) website.

Candidate Predictive Measures

Participants completed a standard intake form that included information on factors known to be prognostic for musculoskeletal pain and/or pertinent to clinical care.^15–18 These factors included age, sex, race, ethnicity, income, employment, education, insurance, geographic region, anatomical region of pain, pain duration, pain onset type, and history of surgery. Comorbidity burden was determined using a composite count of conditions included in the Charlson and Functional Comorbidity Indices.^19,20 These measures were collected only at baseline.

We included a tool developed through the OSPRO development cohort study: the OSPRO Yellow flag (OSPRO-YF).¹⁴ The OSPRO-YF tool includes items from pain vulnerability (negative affect and fear-avoidance) and pain resilience domains (positive affect and self-efficacy) to aid with efficient identification of pain-associated psychological distress and coping.²¹ The OSPRO-YF tool estimates scores for full-length psychological questionnaires and identifies the presence of yellow flags, which are psychological indicators for poor prognosis.¹⁴ We used the 10-item tool in this analysis and followed the simple summary score method, which involves summing all item responses from the tool, with pain resilience items (questions 14, 15, and 17 on the original full-length OSPRO-YF) reverse scored. Simple summary scores range from 3–53 for the 10-item version, with higher simple summary scores indicating higher psychological distress. We have used this method extensively in previous analyses to predict clinical and healthcare utilization outcomes.^{9,10,13,22,23} In those analyses, the 10-item version of the tool performed better than or similarly compared to the longer 17-item version. The OSPRO-YF is valid and reliable, with additional information on psychometric properties and scoring provided in a recent publication by Butera et al.²⁴ This measure was collected at baseline, 4 weeks, 6 months, and 12 months.

Primary Outcome

Pain is known to be a significant driver of seeking healthcare and increased healthcare utilization.²⁵ Pain intensity is one of the most commonly collected pain measures across clinical settings,²⁶ therefore pain intensity was chosen as the primary outcome in the analyses. Pain intensity was assessed with the 0–10 numeric rating scale, and participants rated their current pain intensity as well as their best (lowest) and worst (highest) pain intensity over the past 24 hours.^19,27,28 We used the average of these three measures to establish overall pain intensity. The outcome of interest, persistent pain, was defined as not achieving a 50% reduction in pain intensity from baseline to 12 months. A 50% reduction was selected because it exceeded commonly recommended minimal change criterion (eg, 30% reduction)²⁹ and represented a larger change indicative of successful outcomes.¹² This criterion was used because it represented a stringent categorization, closer in magnitude to patient-centered definitions for pain and disability recovery.^30–32

Data Analysis

Descriptive analyses were conducted for all covariates, including demographic, clinical, and symptom characteristics of the sample at baseline. The simple summary score of OSPRO-YF at baseline and 4-week, 6-month, and 12-month follow-up was calculated.³³ Continuous variables were summarized using mean with standard deviation, median with interquartile range, and range where appropriate. Categorical variables were summarized using frequency with percentage.

For our primary analysis, we developed candidate predictive models and determined which would best inform a risk assessment tool to predict a failure to achieve a 50% reduction in pain intensity at 12 months. We took two approaches to model development. The first was a longitudinal follow-up approach. The model derived from this approach was designed to maximize predictive accuracy and model fit with the most full item set considering all available predictors in the OSPRO cohort, therefore representing the “optimal case” scenario for prediction. The second approach was to develop short-term follow-up models informed by variables commonly measured during most care episodes, as well as OSPRO-YF change score between baseline and 4 weeks only. We compared short-term models to determine the extent to which accuracy was reduced from the optimal statistical case (ie, Full predictor model). All models were fitted for the same complete dataset with 279 subjects.

Longitudinal Model Approach

We built the longitudinal model using a stepwise algorithm with both forward and backward selection that included all candidate predictors: demographic covariates (Table 1), baseline clinical covariates (Table 2). This model included all change scores in the OSPRO simple summary score up to 12 months (baseline score, baseline to 4 weeks, 4 weeks to six months, six months to 12 months). The algorithm terminated when no improvement in model fit according to the Akaike Information Criterion was achieved. This model was used as the comparison standard for the comparison models as we viewed this approach as providing the optimal prediction accuracy given all available predictors.

Table 1 Demographics of OSPRO Validation Cohorta

Table 2 Clinical Characteristics and Pain Intensity Outcomes for OSPRO Validation Cohorta

Comparison Short-Term Predictor Models

The comparison predictor models were built with pain-related variables that could be routinely collected during the clinical encounter chosen a priori based on available variables in the dataset. The predictors selected in the logistic regression models included the presence of previous episodes of musculoskeletal pain, anatomical region of pain, baseline OSPRO-YF simple summary score and 4-week change score, and baseline pain intensity score. Compared to the full predictor model, the reduced predictor models did not include follow-up OSPRO scores past 4 weeks as potential predictors. We developed two different types of reduced predictor set models. The first was a “baseline only” model that included baseline predictors only. The second included baseline predictors plus change in OSPRO-YF summary score between baseline and 4 weeks (ie, “early change” model). The early change model would clarify the prediction value added by considering early change in psychological distress and provide an option to update prognosis based on early response to treatment. However, this model would not be suitable for use at initial evaluation since it would include a 4-week change score. This modeling approach focused on developing models with clinical utility that align with treatment planning; therefore, follow-up time change scores past 4 weeks were not included. Previous analyses have demonstrated only modest improvement in accuracy for clinically oriented models past 4 weeks.³³

Comparison of Models

Estimates of individual parameters and the corresponding area under the curve (AUC), a measure of prediction accuracy, were reported for each model and were compared to determine the ideal model (or models) for use in developing the P3 tool that weighs predictive accuracy and pragmatic implementation considerations. Sensitivity analyses were performed for all models to consider the applicability of our cut-off for reduction in pain intensity. We found that using 30% reduction in pain intensity was similar to the models for 50% reduction in pain; therefore, only models with 50% pain reduction are presented. We reported the model parameter estimates, standard error, and p-values for each model. All analyses were performed using R version v3.7.0. We used pROC 1.15.3 R package for calculating AUC.³⁴

Results

Patient Characteristics

A total of 440 participants completed baseline measures with primary complaints of neck (n = 98, 22.3%), shoulder (n = 107, 24.3%), low back (n = 118, 26.8%), or knee (n = 117, 26.6%) pain. Demographics of the sample are reported in Table 1.

Baseline patient characteristics and 12-month outcomes for pain intensity, opioid use, and surgery are reported in Table 2. A total of 279 (63.4%) participants completed the 12-month follow-up, with primary complaints of neck (n = 59, 21.1%), shoulder (n = 66, 23.7%), low back (n = 72, 25.8%), or knee (n = 82, 29.4%) pain. A previous study on this cohort reported minimal differences between the total participants and the cohort of patients completing all follow-up.²³ We did not conduct imputation for missing data because previous outcomes prediction analyses using the OSPRO cohort have demonstrated no meaningful differences between imputed and complete case analyses.^9,10,23

Statistical Modeling of 50% Reduction in Pain Intensity at 12 Months

After the completion of the stepwise regression, the predictors included in the final longitudinal model were the presence of previous episodes of musculoskeletal pain, anatomical region of pain, baseline OSPRO-YF simple summary score, baseline average pain intensity score, and changes in OSPRO-YF simple summary score from baseline to 4 weeks, 4 weeks to 6 months, and 6 to 12 months. Full model parameters and results of the longitudinal predictor set and comparison short-term predictor set models are reported in Table 3. The longitudinal model yielded an AUC of 0.79. The baseline-only OSPRO model yielded an AUC of 0.68, while the early change model yielded a slightly higher AUC of 0.71 and improved model fit compared to the baseline-only model. Because the lean predictor models achieved acceptable levels of accuracy and model fit, they were used to develop the final versions of the P3 tool.

Table 3 All Model Estimates with Individual Parameters for Predicting 12-Month 50% Pain Reduction

P3 Tool Calculation of Predicted Probability for Not Achieving a 50% Reduction in Pain Intensity

The probability of not achieving a 50% reduction in pain intensity at 12 months can be calculated using the steps outlined in Table 4. Two models inform the tool: the baseline only model with baseline variables that is used at baseline or initial assessment to predict probability of not achieving a 50% reduction in pain intensity, and the early change model that includes baseline-4 week change in OSPRO-YF and is to be used as a follow-up measure at 4 weeks to update prediction probabilities and communicate the change in risk. Inputting patient-specific values into the logistic regression equations will produce the log odds of not achieving a 50% reduction in pain intensity (step 1); the log odds must then be transformed into a probability estimate (step 2) and then the percentage probability of not achieving a 50% reduction in pain intensity at 12 months is estimated (step 3):

Table 4 Personalized Pain Prediction (P3) Tool Manual Calculations

Step 1: Baseline: Log-odds = −0.131 + 0.402 * Anatomical region-Low Back (Y/N) + 1.397 * Anatomical region-Shoulder (Y/N) + 0.771 * Anatomical region- Knee (Y/N) −1.062 * Previous Episodes (Y/N) + 0.100 * Baseline Pain Intensity Score −0.030 * YF 10 Baseline simple summary score

Early Change: Log-odds = 0.069 + 0.522 * Anatomical region-Low Back (Y/N) + 1.483 * Anatomical region-Shoulder (Y/N) + 0.958 * Anatomical region- Knee (Y/N) - 1.069 * Previous Episodes (Y/N) + 0.133 * Baseline Pain Intensity Score - 0.055 * YF 10 Baseline simple summary score + 0.092 * Change in YF 10 Simple summary score baseline - 4 weeks

Step 2: Probability of 50% reduction in Pain= exp(log-odds)/(1 + exp(log-odds))

Step 3: % Probability of Not achieving a 50% reduction in pain intensity = (1-Probability 50% reduction in pain)*100

To demonstrate the use of the P3 tool, three hypothetical patient cases can be found in Table 5. A user-friendly mock-up of the tool in Microsoft Excel is provided in the Supplemental Material.

Table 5 Simulated Patient Cases Using P3 Tool

Discussion

In this study, we examined two modeling techniques to build a multidimensional prediction tool that estimates risk of not achieving a 50% reduction in pain intensity at 12 months after an episode of physical therapy. This analysis adds to existing literature by reporting a tool to predict outcomes defined by change in pain intensity, complimenting predictive approaches that include prolonged disability and chronic or persistent pain states as endpoints.^6,8,10 When examining candidate predictive models in this analysis, the longitudinal model represented the “optimal case” level of predictive accuracy against which the comparison short-term predictor models could be compared. Both baseline only and early change models had lower accuracy than the longitudinal model but were within an acceptable accuracy range for predictive models.^35,36 Compared to the baseline-only predictive model, the early change model demonstrated modestly higher accuracy, suggesting psychological distress monitoring in the first 4 weeks of treatment can improve accuracy of prediction for pain intensity outcomes, but not drastically above assessments that consider baseline values only. Therefore, we used both short-term models to inform the P3 tool so clinicians have the option of using this tool at baseline only or administering it again 4 weeks later to update prognosis.

Serial assessment of risk is not a common paradigm in physical therapy but does hold promise for improving clinical decision-making.³⁷ The additional 4-week assessment of psychological distress using the P3 tool might be most beneficial with those patients demonstrating high risk at baseline to determine whether early responses to treatment has improved the initial prognosis. For those who do not show improvement, a change in the current physical therapy approach to include psychologically informed approaches³⁸ or additional referral for multidisciplinary care might be indicated. The P3 tool using the 4-week change model may also be helpful in determining how much a patient will need to reduce his or her psychological distress to improve the probability of achieving a 50% reduction in pain intensity. For instance, a patient with chronic low back pain who reports 7/10 pain intensity and OSPRO-YF summary score of 35 at baseline would have a 76% probability of not having a 50% reduction in pain intensity at 12 months. With the P3 tool it can be estimated that the patient will need to achieve a 4-week improvement of approximately 20 points in OSPRO-YF summary score to decrease that probability to 40%. These target values could be very helpful in goal-setting, treatment decision-making, and longitudinal monitoring.

The derivation version of the P3 tool has similar statistical accuracy for 12 month pain intensity reduction compared to validation versions of other related risk assessment tools that included disability or pain persistence as endpoints, such as the STarT Back Screening Tool (SBST),⁸ the PICKUP tool,³⁹ and the Pain Belief Screening Instrument.⁴⁰ However, the P3 tool has yet to be externally validated in an independent sample so it is likely that accuracy of the tool may vary. Even when this limitation is considered it is still relevant to consider that the existing validated tools did not report pain intensity endpoints so additional tools, like the P3 tool, still merit further investigation for those interested in predicting pain intensity endpoints. The use of prediction tools like the SBST, the PICKUP tool, and now the P3 provides advantages over clinician judgement alone for predicting outcomes in patients with musculoskeletal pain.³⁹ Similarly, the P3 tool advances risk assessment in musculoskeletal pain by directly considering pain resilience domains (positive affect and self-efficacy) while estimating risk of 12-month pain intensity outcomes. The P3 tool also delivers value by enabling clinicians and their patients to better account for the dynamic nature of prognosis by incorporating early treatment responses and its impact on psychological distress. This capability is not possible with “static” prediction tools that were designed to inform prognosis at a single point in time, typically at the beginning of a care episode.

Strengths and Limitations

Our study’s strengths include that we were able to evaluate the performance of short-term follow-up models against a longitudinal “optimal case” predictor model to inform a risk assessment tool. In addition, we incorporated a brief but comprehensive psychological screening tool to complement other health-related and patient-reported variables in our models. We also acknowledge some limitations readers should consider when interpreting our results. This was further analysis of the OSPRO cohort; therefore, we were limited in the variables collected, the time points at which they were collected, and results can only be generalized to this cohort. The sample size was adequate to perform the planned analyses, but we have yet to validate the models in a separate cohort or validation dataset. Independent validation of this risk assessment tool, comparison to other existing tools, and strategies for clinical implementation, should be the goal of future research. Additional limitations that are important to consider when interpreting these results include the lack of details on the type of physical therapy treatments received and the lack of diagnostic categories. The lack of this information means that this risk calculator cannot be used to generate estimates for specific treatment approaches or specific diagnostic conditions.

Conclusion

We have developed the P3 tool to identify patients at risk for not achieving a 50% reduction in pain intensity 12 months after physical therapy. The P3 tool is a highly clinically useful tool as it provides a feasible approach to concisely predict risk for patients with musculoskeletal complaints seeking care from physical therapists. The P3 tool can be tailored for use at baseline or initial assessment only, or in conjunction with a follow-up assessment at 4 weeks to refine and update the initial prognosis. The P3 tool’s focus on prediction of pain intensity reduction adds to existing tools that have focused on disability or chronic pain outcomes. More research is needed to independently validate the tool in a prospective cohort of patients, compare it with existing tools, and identify best strategies for clinical implementation.