A narrative review of data collection and analysis guidelines for comparative effectiveness research in chronic pain using patient-reported outcomes and electronic health records

Introduction

The Institute of Medicine (IoM) estimates that chronic pain affects over 100 million people in the USA and is associated with over $600 billion in health care costs.¹ Chronic pain is a multifactorial condition that is notoriously difficult to treat effectively. The specific pain and symptoms experienced by each patient are the result of complex interactions between their anatomy, physiology, and biopsychosocial environment.² Although generalized treatment algorithms have proven effective for diseases like hypertension and diabetes mellitus, there are few and possibly no reliable and validated treatment algorithms for chronic pain, and pain medicine practitioners often need to develop individualized care plans for their patients. They must take into consideration the possible responses to a whole range of treatments such as medications, injections, rehabilitation, and mental health care. Thus, knowledge about the real-world effectiveness of treatments for chronic pain is crucial to advancing the field and improving patient care.

In the landmark report, Relieving Pain in America: A Blueprint for Transforming Prevention, Care, Education, and Research, the IoM (now known as the National Academy of Medicine) laid out guidelines for conducting comparative effectiveness research (CER) to develop treatment regimens for chronic pain that optimize benefits to the patients while minimizing the cost of care.¹ CER is a clinical research method in which established medical interventions are directly compared to determine which are the most beneficial and cost-effective.³ The IoM also promoted creating Learning Healthcare Systems where clinical practice is informed by scientific evidence, and reciprocally, clinical research agendas are influenced and informed by patients and providers.

The collection and integration of patient-reported outcomes (PROs) are essential to the Learning Healthcare System model. While electronic health records (EHR– also called electronic medical records) provide a wealth of information about the demographics, diagnoses, and treatments administered to each patient, PROs provide a quantitative measure of how those treatments affect each patient. Combining EHRs with PROs allows researchers and clinicians to perform CER at an unprecedented level of coverage and precision.⁴

The Collaborative Health Outcomes Information Registry (CHOIR) is a PRO tracking system designed specifically for chronic pain patients and pain management programs.⁵ CHOIR is an open-source, web-based software program modifiable at each local installation that was developed by researchers at Stanford University. When the PROs are combined with data extracted from the EHR, this repository provides a powerful tool for CER in chronic pain. This review will outline best practices for conducting CER using PROs and EHRs using the CHOIR system as an example. While this paper specifically focuses on chronic pain and pain management programs, these guidelines are applicable to any complex medical condition and any type of health care system in the USA.

CER using practice-based evidence

CER can be performed with a variety of clinical study designs. Randomized controlled trials (RCTs) have long been considered the “gold standard” of exploratory clinical research, but these studies require significant amount of time, money, and researchers committed to each project. RCTs are also very inefficient; the IoM estimates that it takes 17 years for new knowledge derived from an exploratory RCT to be incorporated into clinical practice.⁶ Observational study designs provide an alternative to RCTs that allow researchers to examine patient populations without manipulating their treatment regimens. While observational studies are usually cheaper and more efficient than RCTs, the quality of these studies is often affected by patient and treatment selection bias, confounding variables, and the use of a heterogeneous subject population.^7,8

Practice-based evidence (PBE, also called Clinical Practice Improvement Methodology) is a model of CER that is designed to minimize common biases in observational studies. In a 2012 paper, Horn et al outlined the requirements for rigorous PBE studies.⁸ In these studies, multidisciplinary and multisite project clinical teams utilize comprehensive age- and disease-specific measures that are implemented as widely as possible and checked regularly for reliability. PBE data are merged into a central study database for analysis and hypothesis testing. Study findings are then implemented into clinical practice for validation testing with the ultimate goal of integration into standard care. PBE studies are designed to improve on traditional observational studies by 1) examining large, diverse patient populations; 2) involving clinicians in the research design and data collection; 3) using carefully selected patient characteristics for analysis to avoid bias; and 4) standardizing data collection and treatment documentation at all research sites. PBE methodology is ideal for conducting “pragmatic” trials that are designed to measure the overall benefit produced by a treatment in a naturalistic clinical setting.⁹ Reviews in the pain medicine literature indicate that such a pragmatic approach based on real-world clinical data may yield CER that can truly improve the treatment of chronic pain.⁹

A PRO database combined with EHRs provides an ideal mechanism for conducting PBE studies and pragmatic trials. The outcomes measured can include quantitative measures such as pain intensity and global health scores as well as qualitative measures like the patient’s impression of change. If outcome measures are collected routinely in clinical practice, researchers are then able to study a large proportion of the clinic population. However, establishing this type of database for PBE research is no small feat and requires a number of considerations for both the EHR and the PRO databases.

EHRs in CER

The Health Information Technology for Electronic and Clinical Health Act was enacted in 2009 as part of the American Recovery and Reinvestment Act to initiate the improvement and implementation of EHR in a wider range of clinical settings.^10,11 While this law has led to greater availability of clinical and administrative data, there are a number of considerations and caveats for combining EHR data with PROs. Hersh et al provided a thorough review of the issues that can arise when using EHR for CER in a 2013 paper.¹² In short, EHR data can be incomplete, inaccurate, and inconsistent. Enterprise data collection and data entry standards can vary greatly both within and between health care systems. It is critical for researchers to understand what data are included in EHRs, to carefully select variables for analysis, and to clearly report the limitations of their analyses as well as any potential problems with their datasets.

The most important consideration when using EHR for any research purpose is that the EHR contains identifiable patient data, including names, dates of birth, medical record numbers (MRNs), addresses, phone numbers, and insurance information, and must be stored according to the Health Insurance Portability and Accountability Act (HIPAA) guidelines. Researchers wishing to use EHR must either 1) obtain informed consent from each patient whose data are included in an analysis, which is often impractical, or 2) obtain a waiver of consent from an Institutional Review Board or research oversight committee. A waiver of consent typically requires that either the research staff have access to EHR data as part of their regular clinical duties or researchers only view data that have been deidentified by an honest broker.^13,14

Accessing EHR data for research purposes can also be a cumbersome task. Most institutions have a research office or committee that can assist with EHR data queries. However, the availability of resources for vetting project requests and obtaining EHR data vary greatly among institutions, which can lead to very long wait times for research data. To mitigate these issues, comparative effectiveness studies that require EHR data should account for when, how, and what EHR data will be acquired during the initial study design phase.

PROs in pain medicine

As stated above, the IoM’s 2011 report emphasized the need to collect PROs for CER.¹ PRO is a catchall term to describe any health-related data provided by the patient, including health and treatment history, symptoms, and lifestyle choices.¹⁵ In 2003, the Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials (IMMPACT) recommended that clinical trials of chronic pain measure six core outcome domains to determine the efficacy and effectiveness of a treatment: 1) Pain, 2) Physical Functioning, 3) Emotional Functioning, 4) participant ratings of Global Improvement, 5) Symptoms and Adverse Events, and 6) Participant Disposition, which includes adherence to the treatment regimen.¹⁶ A comprehensive PRO system should include measures that cover all six IMMPACT domains, but there is not an accepted standard currently for which specific measures should be utilized.¹⁷ While a number of measures and guidelines exist, the specific makeup of a PRO questionnaire can be decided by clinicians, researchers, and/or administrators at each site.

The NIH Patient-Reported Outcomes Measurement Information System (PROMIS) was created in 2004 as part of the NIH Roadmap Initiative.^18,19 The PROMIS initiative established a collaborative network of clinicians, researchers, and measurement experts tasked with developing and validating item banks to measure symptoms and health statuses for a wide range of chronic conditions, including chronic pain. The PROMIS is currently administered by the NIH HealthMeasures (also known as the Person-Centered Assessment Resource) grant, and all measures are available to clinicians and researchers free of charge at www.healthmeasures.net. As of 2018, there are over 300 PROMIS measures covering physical, mental, and social health that are applicable to both the general population and patients with chronic health conditions. PROMIS measures are available as full item banks, computer adaptive tests (CATs), fixed-length short forms, and comprehensive profiles. One advantage of the PROMIS measures is that the percentile scoring is benchmarked to US national population norms, including both diseased and healthy normal individuals. This facilitates comparisons across geographic regions and different health care populations.

For pain medicine CER, the PROMIS measures provide an ideal and effective way to collect PROs in chronic pain patients that cover all six IMMPACT domains (Table 1). Additionally, there are other validated measures available that provide an alternative and/or supplement to the PROMIS measures, including the Brief Pain Inventory,²⁰ the Oswestry Disability Index,²¹ and the PainDETECT Questionnaire for Neuropathic Pain.²² When choosing which measures to include in a PRO database, it is important to balance the breadth and quality of data collected with the burden on the patient. For example, PROMIS CATs use item response theory,²³ so the exact questions administered to each patient depend on that patient’s previous answers and the total number of questions administered to each patient can vary from 4 to 12. Fixed-length short forms provide an alternative to the CATs, so each patient answers the same number of questions. Although this can reduce the maximum number of questions that a patient could be given, T-scores calculated from short forms may not be as precise as those from CATs.^18,19

Table 1 Patient-reported outcome measures for chronic pain patients Notes: Measures used in the UPMC Pain Medicine CHOIR surveys are listed in italics. Abbreviations: CAT, computer adaptive test; CHOIR, Collaborative Health Outcomes Information Registry; COMM, Current Opioid Misuse Measure; DSM-V, Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition; IMMPACT, Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials; PROMIS, Patient-Reported Outcomes Measurement Information System; PTSD, post-traumatic stress disorder; SOAPP, Screener and Opioid Assessment for Pain Patients; UPMC, University of Pittsburgh Medical Center.

For a PRO system to be effective for CER and useful in clinical practice, it is critical that 1) the collection of outcome data does not interfere with normal clinical operations, 2) the specific measures used are of interest to clinicians and researchers, and 3) the patients are engaged with and not unduly burdened by the questionnaires. It has been well-established that as questionnaire length increases, both the rate and quality of responses will decrease.^24,25 While it is tempting to collect a wide variety of data to assess every variable of interest, it is important to understand that both data quality and response rates will be optimized when the questionnaire length is kept to a minimum. Patient burden can also be reduced by increasing the accessibility of the questionnaires,²⁶ by offering multiple options for survey completion (such as via e-mail or text messaging), offering surveys in multiple languages, and providing options for patients who are vision impaired, have low literacy levels, or are unable to use a computer or touchscreen.²⁷

A PRO system requires both motivated patients and motivated providers to ensure its success. The patient must feel that the data they supply at each appointment are important to their care and are actually being used by their provider. If a patient is able to see that their survey information is being used to directly guide their care, they are more likely to complete the survey and provide high-quality responses to each question.²⁵ This approach has also been noted to improve patient satisfaction with care scores.²⁸ It is important for providers to acknowledge and thank the patient for completing the survey and, if needed, to go over individual scores and answers. However, a potential downside of this enhanced patient engagement approach is that patients may be motivated to exaggerate their improvement to please their physician or exaggerate their deterioration to have additional treatments prescribed. The field is lacking good studies on the potential bias due to social desirability in completing PROs. Nevertheless, the ultimate goal when collecting PROs is to continually involve the patients in their own care and to take their needs, preferences, and priorities into account when making shared treatment decisions.

Patient outcomes repository for treatment (PORT) at the University of Pittsburgh

The data repository system at the University of Pittsburgh Medical Center (UPMC) joins PRO collection via CHOIR with extraction and crossmatching of individual patient-level data from the EHR. CHOIR is an open-source, web-based PRO software specifically designed for pain medicine programs, but applicable to a range of disease areas.^5,29–32 The CHOIR system includes PROMIS CATs and short forms, other validated measures, and site-specific customizable surveys.

Data collection

CHOIR outcomes are collected via surveys that patients are asked to complete at each clinic visit. To increase completion rates and accessibility, patients are offered three options for survey completion. First, a few days before their scheduled appointment, patients are sent an email that contains a link to that appointment’s CHOIR survey. Since CHOIR is entirely web based, patients can complete the survey before their scheduled appointment on their own computers or mobile devices. Second, if the patients do not complete the survey before their appointment, they have the option to complete the survey on a clinic-owned tablet in the waiting room. Third, patients who do not wish to use a tablet have the option of completing the surveys on paper. It should be noted that paper versions of CHOIR surveys need to be manually entered into the CHOIR database by clinic staff and need to be stored according to HIPAA guidelines.

It is common for patients receiving injections to return to a pain clinic after the injection has worn off. In such cases, the surveys collected at the patient’s injection and follow-up appointments may show the same pain levels. Although the patient may have gotten significant relief from the injection, the pain scores collected at each appointment would suggest that the injection was not effective. To address this, a very brief survey (taking <2 minutes to complete) is emailed to patients every 6 weeks if they do not have a follow-up appointment within this time frame. Collecting these nonappointment follow-up surveys enables CER by obtaining PROs more frequently to improve the analyses of treatment effectiveness. A weblink to contact the office with any questions or concerns is also included with the nonappointment survey request. This feature enables remote patient monitoring to assist in population-based chronic disease management.³³

The CHOIR software was designed to be flexible so that each site could customize its questionnaires. Early versions of CHOIR allowed for two survey types: initial surveys for new patients and returning patients answering their first CHOIR survey and shorter follow-up surveys for all other appointments. Newer versions of the CHOIR software can create customized surveys for different appointment types such as procedures and psychological evaluations. In 2016, UPMC Pain Medicine implemented the CHOIR system into its eight regional clinics in western Pennsylvania. Using the flexible software architecture, UPMC customized initial and follow-up surveys to include PROMIS measures and survey questions from the original Stanford CHOIR surveys plus new measures and custom site-specific questions. Measures used in the UPMC version of the CHOIR system are italicized in Table 1, and the UPMC data dictionary does not include all of the PROMIS item banks described. More specifically, the UPMC CHOIR surveys have added the PainDETECT neuropathic pain survey,²² a validated Opioid Adherence Checklist to identify self-reported misuse,³⁴ and two custom 5-point Likert scale “Impression of Change” questions (similar to the Patient Global Impression of Change scale)³⁵ that ask the patient how much improvement they have noticed since they first started coming to the pain clinic and for the most recent treatment they have received. The Impression of Change measure is a valuable tool because it adds a subjective patient-reported assessment alongside the more objective validated PROMIS measures. This is valuable, for example, in a patient who had an intervention and benefited from it and is now returning to the clinic because the previously beneficial effects of the intervention have waned. If a clinician or researcher was to look only at PROMIS pain intensity scores for that patient, it would appear that the patient’s pain remained the same and there was no improvement. However, that same patient may report that they have seen meaningful improvement since they started coming to the pain center.

One concern often raised when gathering PROs is whether the data are more valid if the patients themselves complete the surveys without any assistance. However, a recent study has indicated that proxy-completed PROMIS measures are highly reliable in comparison with the patient-completed measures.³⁶

Clinical uses of CHOIR

When patients complete a CHOIR survey in the waiting room or while they are waiting for the physician to come into the examination room, their answers are used to generate a custom PDF report for their providers. These can be reviewed prior to seeing the patient and can be reviewed with patients in the examination room (Figure 1). These reports contain an image of the body map showing where the patient is experiencing pain, PROMIS T-scores, calculated scores from other measures, and a list of each question given and the patient’s corresponding answers. Follow-up survey reports contain line graphs that plot the change in scores over time. This allows clinicians to easily see how the patient’s pain, mental, and physical health have changed over time and in response to different treatments. This PDF report can then be pasted into the encounter notes in the patient’s EHR. Future releases of EPIC (the EHR at UPMC and many other institutions) will enable the discrete variable data from the PROs and the PDF reports to be automatically transmitted from the CHOIR database to EPIC. The presentation format of the data to patient and provider is frequently reviewed through qualitative analysis of periodic semistructured interviews to determine if the clinical information is conveyed effectively. For example, in the future, a more refined “dashboard” presentation with bar graphs may be more effective than the existing PDF reports.

Figure 1 Example of a patient PDF report generated by the CHOIR system. Abbreviation: CHOIR, Collaborative Health Outcomes Information Registry.

A key strength of the CHOIR system is that data are directly reported and entered by the patients themselves and not based on or subject to the physician’s report or documentation. This avoids interpretation or reporting bias that may be present in other systems. More importantly, this serves to integrate the patient into their own treatment team, which can strengthen the bond and enhance trust between patient and providers. This is particularly true when the clinician emphasizes to the patient that they reviewed their CHOIR report. This is an invaluable benefit of the CHOIR system, as the doctor–patient relationship can be difficult to establish in pain management because patients have often seen multiple specialists and had numerous unsuccessful treatments before they visit a pain medicine physician.

PORT database architecture

All CHOIR data collected from clinic patients are stored in a custom PRO database managed by the organization’s Information Services Division. This database is stored and managed separately from the outpatient EHR database. Patient data in the CHOIR database can be linked to EHR data by matching MRNs, encounter dates, and provider information. The CHOIR software contains a suite of tools to monitor data quality, including the number of completed and partially completed surveys.

Access to the outpatient EHR database is set up with a customized view filter so that all relevant EHR data (and no unnecessary data) are available to researchers and clinicians. The available EHR data include 1) encounter information for all pain medicine clinic patients at pain medicine clinic appointments and medical history information for these patients, including 2) medications ordered, 3) procedures ordered, 4) medical and surgical history, and 5) medical problem list. The same functionality applies to the inpatient EHR database at UPMC, which can be cross-linked to visit specific data from the outpatient EHR and the CHOIR PROs. Thus, health care utilization information, such as emergency department visits or hospital admissions, is available in a robust fashion. At the University of Pittsburgh and UPMC, this combined PRO and EHR data repository is referred to as the PORT. Figure 2 illustrates the processes required to generate predictive analytics from PROs and EHR data. Inputs of data are performed by patients, providers, and health care staff. Examples of data that can be collected via PRO and EHR are listed in Figure 2. PRO and EHR data are then fed into a shared database and linked via common patient identifiers such as MRNs. Data queried from the linked database must undergo some form of data processing to prepare for analysis. This data processing can include deidentification, categorization of variables (such as ages converted to age ranges), case–control matching, and/or feature selection to remove irrelevant and redundant variables from the model. Variables with predictive validity are then analyzed as covariates in the final predictive data analysis. Data processing should also identify the outcome variables to be used in the predictive data analysis, such as responses to treatments, clinically significant improvement in pain and/or function, or tapering of opioid medication.

Figure 2 Schematic representation of the processes required to generate predictive analytics from patient-reported outcome and electronic health record data.

At UPMC, the CHOIR PROs and outpatient and inpatient EHR information are stored in Oracle servers and comprise the PORT data repository. Data queries must be programmed in SQL and performed with SQL-based database software such as SQLDeveloper. This requires specific coding skills that may be difficult for physicians and researchers who do not have backgrounds in programming or data analysis. To address this, our research group at UPMC and the University of Pittsburgh has developed a data request form that allows clinicians and researchers to receive CHOIR and EHR data without needing database access or specialized programming skills. Data requests are vetted and fulfilled by research administrators and database managers.

Research with PORT

The PORT platform includes the integration of the CHOIR system into pain medicine clinics and linkage of its database to outpatient and inpatient EHR data. It provides an ideal data source to conduct high-quality practice-based CER in a Learning Healthcare System. With the PROMIS measures included in CHOIR and the customized patient PDF report for each survey, clinicians and researchers are able to evaluate each patient with tools that are both normalized and validated for the US population. Continuous collection of these measures at each appointment greatly facilitates personalization of treatment plans and evaluation of prior interventions. As of this publication date, we have collected over 60,000 surveys in >24,0000 unique patients, with each patient’s PRO data cross-linked to their corresponding EHR data. On average, CHOIR data are captured in 70% of our patients. New surveys are accrued at a rate of approximately 500 per week.

There are many considerations regarding statistical analysis of PRO and EHR data for CER. Overall, CER must be based on valid data and sound analyses. Before data analysis can occur, researchers must select appropriate sample sizes and analysis methods for the research question, address the psychometric properties of the PROs, and perform data cleaning, formatting, and harmonization where necessary.^37,38

Making causal inferences from observational data, even when gathered in a high quality and consistent fashion, is plagued by a myriad of unmeasured confounders.³⁹ One of the most important is “confounding by indication,” which refers to the nonrandom choice of the specific treatment by the provider. In other words, providers prescribe a certain treatment for a patient based on their unique presenting characteristics, which may not be measured or cannot be captured by the PRO or EHR data used for analysis. This introduces an inherent level of bias to CER because treatments are not being compared with each other between patient groups whose characteristics are equal (apples to oranges).⁴⁰

Multiple statistical procedures exist to minimize the effects of this confounder and many other unmeasured confounders. Some of the most recommended are multivariate regression modeling and using propensity scores.⁴¹ Propensity scores narrow the patient populations for the treatments being compared statistically to those with comparable characteristics at baseline.⁴² However, the variables used for matching at baseline cannot also be outcomes of interest (such as pain intensity levels), and thus, successful propensity score matching will not create groups of patients that have identical baseline characteristics as a whole. Least absolute shrinkage and selection operator variable selection methods comparing the quartiles of propensity scores between groups and using multiple propensity scores can improve the performance of propensity score matching in the causal inference models.^43–45 In particular, multiple propensity scores have shown to be effective in comparing multimodal combinations of treatments in musculoskeletal disease.⁴⁴

Research with CHOIR

The CHOIR system has already been used to conduct several innovative research studies. Researchers at Stanford, where the CHOIR software was developed, have published at least 15 articles to date that utilized CHOIR data. Most of these studies used cross-sectional path modeling to examine the relationships between several PROMIS pain, mental, and physical health measures. These studies found that fatigue is a key mediator of physical function and pain interference⁵ and that satisfaction with social roles is an important driver of the relationships between physical function, depression, anger, and pain.²⁹ Another cross-sectional study showed that perceived injustice has a strong correlation with anger in chronic pain patients, and this relationship is at least partially mediated by social satisfaction.³⁰ Perceived injustice also appears to contribute to greater use of prescription opioids, increased pain intensity, and decreased physical function and pain acceptance.^32,46 Using similar methods, researchers also found that prescription opioid use is related to pain catastrophizing and pain intensity, and at low pain catastrophizing levels, women are more likely than men to have an opioid prescription.⁴⁷

Stanford researchers and clinicians have also developed a Pediatric-Collaborative Health Outcomes Information Registry (Peds-CHOIR) to improve treatment and facilitative research in children with chronic pain.³¹ Examining both adult and pediatric chronic pain populations has allowed researchers to discover how chronic pain changes with age. For example, one cross-sectional modeling study found that the relationship between pain catastrophizing and pain interference is strongest in adolescents but decreases with age.⁴⁸

These groundbreaking CHOIR research studies conducted in active pain medicine clinics have allowed clinicians and researchers to examine, model, and understand chronic pain in ways that are simply not possible with traditional clinical research studies. However, there has not been a great deal of CER conducted using CHOIR data. This paper has outlined why the CHOIR PROs system combined with EHRs provide an excellent tool for CER in pain medicine. By examining existing patient populations and tracking how they respond to their treatments, clinicians and researchers can answer many “big” questions about pain, such as which treatments work for which patients, which treatments do more harm than good, and how do we define when a patient is “better”?

Conclusion

The combination of PROs and electronic health data is an innovative and exciting mechanism to conduct CER in chronic pain. These tools will allow pain medicine providers to more easily and accurately develop individualized treatment plans for their patients. Patients, in turn, will be empowered to be directly involved in their own care and to take ownership of treatment decisions. This can improve the patient-provider relationship and ultimately lead to better and more effective care in pain medicine.

Disclosure

The authors report no conflicts of interest in this work.

References

1.		Institute of Medicine of the National Academies Committee on Advancing Pain Research Care and Education Board on Health Sciences Policy. Relieving Pain in America: A Blueprint for Transforming Prevention, Care, Education, and Research. Washington, DC: The National Academies Press; 2011.
2.		Dubois MY, Gallagher RM, Lippe PM. Pain medicine position paper. Pain Med. 2009;10(6):972–1000.
3.		Institute of Medicine of the National Academies Committee on Comparative Effectiveness Research Prioritization Board on Health Care Services. Initial National Priorities for Comparative Effectiveness Research. Washington, DC: The National Academies Press; 2009.
4.		Franklin PD, Ayers DC, Berliner E. The essential role of patient-centered registries in an era of electronic health records. NEJM Catalyst; 2018. Available from: https://catalyst.nejm.org/force-tjr-patient-centered-registries/. Accessed November 10, 2018.
5.		Sturgeon JA, Darnall BD, Kao MC, Mackey SC. Physical and psychological correlates of fatigue and physical function: a Collaborative Health Outcomes Information Registry (CHOIR) study. J Pain. 2015;16(3):e291–e298.
6.		Institute of Medicine Committee on Quality of Health Care in America. Crossing the Quality Chasm. Washington, DC: National Academy Press; 2001.
7.		Dreyer NA, Tunis SR, Berger M, Ollendorf D, Mattox P, Gliklich R. Why observational studies should be among the tools used in comparative effectiveness research. Health Aff. 2010;29(10):1818–1825.
8.		Horn SD, Dejong G, Deutscher D. Practice-based evidence research in rehabilitation: an alternative to randomized controlled trials and traditional observational studies. Arch Phys Med Rehabil. 2012;93(8 Suppl):S127–S137.
9.		Rowbotham MC, Gilron I, Glazer C, et al. Can pragmatic trials help us better understand chronic pain and improve treatment? Pain. 2013;154(5):643–646.
10.		Gold M, Mclaughlin C. Assessing HITECH implementation and lessons: 5 years later. Milbank Q. 2016;94(3):654–687.
11.		Stark P. Congressional intent for the HITECH Act. Am J Manag Care. 2010;16(12 Suppl HIT):SP24–SP28.
12.		Hersh WR, Weiner MG, Embi PJ, et al. Caveats for the use of operational electronic health record data in comparative effectiveness research. Med Care. 2013;51(8 Suppl 3):S30–S37.
13.		Wade TD, Zelarney PT, Hum RC, Mcgee S, Batson DH. Using patient lists to add value to integrated data repositories. J Biomed Inform. 2014;52:72–77.
14.		Shin SY, Kim WS, Lee JH. Characteristics desired in clinical data warehouse for biomedical research. Healthc Inform Res. 2014;20(2):109–116.
15.		Howie L, Hirsch B, Locklear T, Abernethy AP. Assessing the value of patient-generated data to comparative effectiveness research. Health Aff. 2014;33(7):1220–1228.
16.		Turk DC, Dworkin RH, Allen RR, et al. Core outcome domains for chronic pain clinical trials: IMMPACT recommendations. Pain. 2003;106(3):337–345.
17.		Porter ME, Larsson S, Lee TH. Standardizing patient outcomes measurement. N Engl J Med. 2016;374(6):504–506.
18.		Cella D, Gershon R, Lai JS, Choi S. The future of outcomes measurement: item banking, tailored short-forms, and computerized adaptive assessment. Qual Life Res. 2007;16(Suppl 1):133–141.
19.		Cella D, Yount S, Rothrock N, et al. The Patient-reported outcomes measurement information system (PROMIS): progress of an NIH roadmap cooperative group during its first two years. Med Care. 2007;45(5 Suppl 1):S3–S11.
20.		Keller S, Bann CM, Dodd SL, Schein J, Mendoza TR, Cleeland CS. Validity of the brief pain inventory for use in documenting the outcomes of patients with noncancer pain. Clin J Pain. 2004;20(5):309–318.
21.		Fairbank JC, Pynsent PB. The oswestry disability index. Spine. 2000;25(22):2940–2953.
22.		Freynhagen R, Baron R, Gockel U, Tölle TR. painDETECT: a new screening questionnaire to identify neuropathic components in patients with back pain. Curr Med Res Opin. 2006;22(10):1911–1920.
23.		Gershon R, Rothrock NE, Hanrahan RT, Jansky LJ, Harniss M, Riley W. The development of a clinical outcomes survey research application: assessment center. Qual Life Res. 2010;19(5):677–685.
24.		Burchell B, Marsh C. The effect of questionnaire length on survey response. Qual Quant. 1992;26(3):233–244.
25.		Bell ML, Fairclough DL. Practical and statistical issues in missing data for longitudinal patient-reported outcomes. Stat Methods Med Res. 2014;23(5):440–459.
26.		Cook KF, Kallen MA, Buckenmaier C, et al. Evaluation of the validity and response burden of patient self-report measures of the pain assessment screening tool and outcomes registry (PASTOR). Mil Med. 2017;182(7):e1851–e1861.
27.		Axén I, Bodin L, Bergström G, et al. The use of weekly text messaging over 6 months was a feasible method for monitoring the clinical course of low back pain in patients seeking chiropractic care. J Clin Epidemiol. 2012;65(4):454–461.
28.		Freel JBJ, Hanmer J. Better physician ratings when discussing PROs with patients. NEJM Catalyst. Available from: https://catalyst.nejm.org/ratings-patients-discussing-pros/. Accessed June 20, 2018.
29.		Sturgeon JA, Dixon EA, Darnall BD, Mackey SC. Contributions of physical function and satisfaction with social roles to emotional distress in chronic pain: a Collaborative Health Outcomes Information Registry (CHOIR) study. Pain. 2015;156(12):2627–2633.
30.		Sturgeon JA, Carriere JS, Kao MJ, Rico T, Darnall BD, Mackey SC. Social disruption mediates the relationship between perceived injustice and anger in chronic pain: a collaborative health outcomes information registry study. Ann Behav Med. 2016;50(6):802–812.
31.		Bhandari RP, Feinstein AB, Huestis SE, et al. Pediatric-Collaborative Health Outcomes Information Registry (Peds-CHOIR): a learning health system to guide pediatric pain research and treatment. Pain. 2016;157(9):2033–2044.
32.		Carriere JS, Martel MO, Kao MC, Sullivan MJ, Darnall BD. Pain behavior mediates the relationship between perceived injustice and opioid prescription for chronic pain: a collaborative health outcomes information registry study. J Pain Res. 2017;10:557–566.
33.		Sargious A, Lee SJ. Remote collection of questionnaires. Clin Exp Rheumatol. 2014;32(Suppl 85):S-168–S-172.
34.		Jamison RN, Martel MO, Huang CC, Jurcik D, Edwards RR. Efficacy of the opioid compliance checklist to monitor chronic pain patients receiving opioid therapy in primary care. J Pain. 2016;17(4):414–423.
35.		Dworkin RH, Turk DC, Wyrwich KW, et al. Interpreting the clinical importance of treatment outcomes in chronic pain clinical trials: IMMPACT recommendations. J Pain. 2008;9(2):105–121.
36.		Alvarez-Nebreda ML, Heng M, Rosner B, et al. Reliability of proxy-reported patient-reported outcomes measurement information system physical function and pain interference responses for elderly patients with musculoskeletal injury. J Am Acad Orthop Surg. Epub 2018 Sep 25.
37.		Pirracchio R, Cohen MJ, Malenica I, et al. Big data and targeted machine learning in action to assist medical decision in the ICU. Anaesth Crit Care Pain Med. Epub 2018 Oct 16.
38.		Vetter TR, Cubbin C. Psychometrics: trust, but verify. Anesth Analg. 2019; 128(1):176–181.
39.		Mcclimans L. The role of measurement in establishing evidence. J Med Philos. 2013;38(5):520–538.
40.		Cnossen MC, van Essen TA, Ceyisakar IE, et al. Adjusting for confounding by indication in observational studies: a case study in traumatic brain injury. Clin Epidemiol. 2018;10:841–852.
41.		Blommestein HM, Franken MG, Uyl-de Groot CA. A practical guide for using registry data to inform decisions about the cost effectiveness of new cancer drugs: lessons learned from the PHAROS registry. Pharmacoeconomics. 2015;33(6):551–560.
42.		Stuart BL, Grebel LE, Butler CC, Hood K, Verheij TJM, Little P. Comparison between treatment effects in a randomised controlled trial and an observational study using propensity scores in primary care. Br J Gen Pract. 2017;67(662):e643–e649.
43.		Freemantle N, Marston L, Walters K, Wood J, Reynolds MR, Petersen I. Making inferences on treatment effects from real world data: propensity scores, confounding by indication, and other perils for the unwary in observational research. BMJ. 2013;347:f6409.
44.		Bergstra SA, Winchow LL, Murphy E, et al. How to treat patients with rheumatoid arthritis when methotrexate has failed? The use of a multiple propensity score to adjust for confounding by indication in observational studies. Ann Rheum Dis. Epub 2018 Oct 16.
45.		Shortreed SM, Ertefaie A. Outcome-adaptive LASSO: variable selection for causal inference. Biometrics. 2017;73(4):1111–1122.
46.		Carriere JS, Sturgeon JA, Yakobov E, Kao MC, Mackey SC, Darnall BD. The impact of perceived injustice on pain-related outcomes: a combined model examining the mediating roles of pain acceptance and anger in a chronic pain sample. Clin J Pain. 2018;34(8):739–747.
47.		Sharifzadeh Y, Kao MC, Sturgeon JA, Rico TJ, Mackey S, Darnall BD. Pain catastrophizing moderates relationships between pain intensity and opioid prescription: nonlinear sex differences revealed using a learning health system. Anesthesiology. 2017;127(1):136–146.
48.		Feinstein AB, Sturgeon JA, Darnall BD, et al. The Effect of pain catastrophizing on outcomes: a developmental perspective across children, adolescents, and young adults with chronic pain. J Pain. 2017;18(2):144–154.