Serious Issues in Authorship, Design, and Conclusions of JAMA Neurology Real-World Evidence Study on Spinal Cord Stimulation Outcomes and Costs as Compared to Conventional Medical Therapy

We read with interest Long-term Outcomes in Use of Opioids, Nonpharmacologic Pain Interventions, and Total Costs of Spinal Cord Stimulators Compared with Conventional Medical Therapy for Chronic Pain, published online in JAMA Neurology in December 2022, and identified numerous serious flaws which make it more misleading than informing.¹ Using real-world evidence (RWE) to probe the utility and cost-effectiveness of medical treatments and devices will potentially affect healthcare policies worldwide and can be a highly effective clinical decision-making tool. As with any clinical study however, bias, false assumptions, misrepresentations, missing data, and study design all play extremely important roles in determining the outcomes. It is with this in mind that the following concerns about limitations of this study must be considered.

The author group consists of two academic cardiologists and numerous employees of Optum Health (UHC); it is unclear who devised the original idea or obtained funding for the study. That the authors were not pain specialists contributed to misunderstanding the data in important contextual ways. Given that the guideline recommends spinal cord stimulation (SCS) only after several different first-line therapies are attempted, it is unlikely that this design adequately accounts for the typical period of time that SCS patients experience chronic pain prior to treatment. Previous RCTs have demonstrated the length of time from chronic pain diagnosis to initiation of SCS of 12 years.² Most SCS patients have failed conventional medical management (CMM) prior to initiation of SCS. The design of this study compares SCS patients refractory to CMM to those undergoing CMM, who are likely to be early (6 months) in the CMM algorithm of treatment. With this comparison, it is to be expected that responders in both groups would eventually experience a reduction in some of the pain “surrogates” identified as study outcomes. This important point is neglected in the JAMA Neurology article. Figure 1 was adapted from the data in Table 1 and 2 of the JAMA Neurology article, illustrating the percentage of patients who underwent epidural steroid injections (ESIs) at baseline, 1–12M and 13–24M, respectively.¹ The conclusion in the text of the article is that there is no difference between the two groups in the 13–24M timeframe, yet the percentage of patients receiving ESIs decreased from 40% to 25% in both groups. This appears to demonstrate that patients in the SCS group, who have previously failed CMM, and patients early in the CMM treatment algorithm are both responding, which is not the conclusion drawn by the authors.

Figure 1 Percent of patients receiving epidural injections. Note: Data from Dhruva et al.1

The study design is unclear, and Figure 2 in the JAMA Neurology article (the Appendix)¹ illustrates an entirely different study design than that described in the text in the Treatments section, with alternate index start dates (trial versus implant). This design confounds the fundamentals of propensity matching by violating the assumption that “treatment allocation must temporally precede the outcome of interest; this temporal sequence is fundamental to avoid reverse causation.”³ In other words, the authors began the 2-year SCS clock with the trial, allowing up to 365 days to provide implantation, but crediting that entire 24-month period to the SCS group. The second tenet of propensity matching that was violated is “exchangeability”, which would have necessitated matching chronic opioid use. The authors failed to account for the possibility that the SCS patients had higher rates of chronic opioid use prior to enrolling in the study (a result of SCS patients being required to “fail” CMM). This was particularly evident during the period of the study, 2016–2019, which coincided with changes in CDC guidance regarding opioid prescribing. This shift in the opioid zeitgeist likely resulted in significant confounds, resulting in over-interpreting opioid prescribing data in a causal manner. The data demonstrate that in the first year, the SCS patients were more likely to be using high dosage opioids, but were no more likely to be using high-dosage opioids in the second year. This indicates that the SCS patients were reducing their opioid intake over time and contradicts the dire conclusion of the study that SCS had no effect on opioid use. In addition, the study contains insufficient detail to understand the specific pain condition for which these individuals were using opioids. Many patients with chronic pain have more than one area of pain (eg, neck and back), and this study is thus limited by its inability to indicate whether the opioid use was for the area of pain being treated with SCS. Likewise, we do not know whether the interventional pain procedures and spinal surgeries that these patients received were for the same pain indication as the SCS was targeting.

It is standard practice to report the costs associated with an intervention and other healthcare costs separately. That the authors failed to do so is quite unusual and certainly curious. How was total cost of care estimated? In the Outcomes section, the authors state, “Total costs of care (actual) were also assessed”, but under the Statistical Analyses section, the authors state, “Total costs of care were modeled using generalized linear models with a gamma distribution and log link.” The authors account for the cost of the SCS, but it is unclear whether they used actual or modeled costs, or some combination of the two. The discussion section content pertaining to complications cites a lay press article and a consumer watchdog organization story, certainly not a discussion fitting for a scientific manuscript driven by data or meaningful analytics. In fact, increased cost was primarily seen in the first year (when surgery and device implant occur). There was no overall difference in health care costs in year 2. If this study had used a longer timeframe, we likely would have seen an overall decline in SCS costs relative to the CMM group.

The JAMA Neurology analysis is flawed in its evaluation of complications. The article makes a point regarding the high rate of device-related complications. While the authors note that 17.9% of the patients experienced a complication in the 2 years following device implantation, the rates for each complication listed are very much in line with other published studies. For instance, the infection rate of 2.1% is rather low given the nature of the device implant. Moreover, the rate of lead or generator “displacement” was only 1.8%. It is difficult to fully understand the reasons behind issues such as lead and generator removals. Was this for revision to a different location? If these procedures were extracted from the database by CPT code, the CPT code for removal of a SCS generator is revision or removal, thus complicating matters if this is how those patients were identified.

In addition, there are major flaws evident in the discussion and evaluation of functional outcomes. SCS is not a stand-alone treatment for chronic pain, but rather should be used as an element of a comprehensive pain treatment plan. Chronic pain ideally involves multiple treatment modalities (medical, psychological, interventional) provided concomitantly for optimal outcomes. What this study cannot show us is whether the patients who received SCS were more functional with the device as one component of their pain treatment (which potentially may continue to include, for example, injections, medications, and physical therapy).

In summary, this RWE analysis is misleading due to lack of authorship with knowledge of the treatment under study, inadequate propensity matching criteria, unclear conflicting study design (figure versus text), enrollment date bias due to shifting opioid guideline and prescribing patterns, misleading conclusions and representations of the data, and unclear conflicting analytical methods. Finally, these “surrogates” of pain are not a validated method of determining effectiveness, making this study fundamentally flawed, at best, and potentially understating the benefits of what has been demonstrated to be an extremely beneficial therapy when applied judiciously.