The use of random-effects models to identify health care center-related characteristics modifying the effect of antipsychotic drugs
Received 2 July 2017
Accepted for publication 11 September 2017
Published 14 December 2017 Volume 2017:9 Pages 689—698
Checked for plagiarism Yes
Review by Single anonymous peer review
Peer reviewer comments 2
Editor who approved publication: Professor Henrik Sørensen
Clementine Nordon,1 Constance Battin,1 Helene Verdoux,2 Josef Maria Haro,3 Mark Belger,4 Lucien Abenhaim,1 Tjeerd Pieter van Staa5
On behalf of the IMI GetReal WP2 Group
1Epidemiological Research, Analytica LASER, Paris, 2Population Health Research Center, Team Pharmaco-Epidemiology, UMR 1219, Bordeaux-2 University, INSERM, Bordeaux, France; 3Parc Sanitari Sant Joan de Deu, CIBERSAM, University of Barcelona, Barcelona, Spain; 4Eli Lilly and Company Limited, Erl Wood Manor, Windlesham, 5Farr Institute, University of Manchester, Manchester, UK
Purpose: A case study was conducted, exploring methods to identify drugs effects modifiers, at a health care center level.
Patients and methods: Data were drawn from the Schizophrenia Outpatient Health Outcome cohort, including hierarchical information on 6641 patients, recruited from 899 health care centers from across ten European countries. Center-level characteristics included the following: psychiatrist’s gender, age, length of practice experience, practice setting and type, countries’ Healthcare System Efficiency score, and psychiatrist density in the country. Mixed multivariable linear regression models were used: 1) to estimate antipsychotic drugs’ effectiveness (defined as the association between patients’ outcome at 3 months – dependent variable, continuous – and antipsychotic drug initiation at baseline – drug A vs other antipsychotic drug); 2) to estimate the similarity between clustered data (using the intra-cluster correlation coefficient); and 3) to explore antipsychotic drug effects modification by center-related characteristics (using the addition of an interaction term).
Results: About 23% of the variance found for patients’ outcome was explained by unmeasured confounding at a center level. Psychiatrists’ practice experience was found to be associated with patient outcomes (p=0.04) and modified the relative effect of “drug A” (p<0.001), independent of center- or patient-related characteristics.
Conclusion: Mixed models may be useful to explore how center-related characteristics modify drugs’ effect estimates, but require numerous assumptions.
Keywords: schizophrenia, effect modification, effectiveness, health care system, hierarchical model
Plain language summary
The generalizability of clinical trial results may be compromised by a selection bias toward key effect modifiers, at a patient- or health care setting level. However, very few studies explored how health care center-level characteristics may modify drugs’ effect estimates. In this case study, a mixed linear regression model was used to explore drugs’ effect modifiers, while also estimating the extent of variance in patients’ outcome explained by unmeasured confounding at a center level. Nearly, one-quarter of the variance at a patient level was explained by unmeasured confounding at a center level, emphasizing the importance of health care setting characteristics. The identification of drugs’ effect modifiers at a center level may contribute to fine-tuning trial design and identifying adequate recruiting centers.
The pharmacological effect of drugs – referred to as the “efficacy” – is best measured using randomized controlled trials (RCTs) conducted under highly controlled conditions. However, it has been acknowledged that RCTs do not always render drugs’ “effectiveness,”1 that is, their effect when delivered in real-world settings, by practicing staff, to broader and more representative populations.2,3 Effectiveness is being increasingly recognized to be of utmost importance for clinical or regulatory decision-making.4–6 Similarly, safety outcomes may be underestimated in RCTs compared to those measured in routine clinical practice.7 The chiasm, or gap, between the “efficacy” of drugs and their “effectiveness,” known as the “efficacy–effectiveness gap,” has been the subject of extensive study over the past decade.8 In particular, studies conducted on the subject aimed at better understanding why, and under which circumstances, RCTs may fail to reflect the impact of drugs on patients treated under routine clinical care and how to overcome this issue. Conceptually, effectiveness may be understood as a result of interactions between the drug’s pharmacological effect and characteristics related to patients (eg, age, gender, comorbidity, and genetics), the genuine use of the drug in routine clinical care (eg, dosage and drug adherence), and the setting of care (eg, access to care and health care cost coverage). From a statistical standpoint, it has been highlighted that population sampling errors occurring in RCTs, known to affect the distribution of important drug effect modifiers, are likely to have an effect upon the investigational drug’s effect estimates.9 Drug effect modifiers are factors (related to the patient, the drug use patterns, or the health care system) for which the effect of the drug is different depending on it being present or not, which is different from confounding. In other words, if an RCT includes a sample of patients who are not representative of the target population “owing to” important drug effect modifiers, then the drug’s effect estimates are likely to be biased. The influence of sampling errors involving drug effect modifiers, upon the risk for an efficacy–effectiveness gap, was evoked in previous publications.10,11 According to Huybrechts et al,12 “the relevant issue is whether the factors that distinguish studied groups from other groups somehow modify the effect in question.” In consequence, it seems important to explore and identify these effect modifiers ahead of designing the trial, so as to define the trial population that would best capture the effect modifiers for the drug under investigation. Of note, these drug effect modifiers may differ from one therapeutic class, or group of drugs, to another.
In the realm of the European public–private GetReal consortium,13 a series of case studies have been performed to explore methods aiming at identifying key effect modifiers of drugs at a patient level, and across several therapeutic areas: diabetes,14 Hodgkin’s lymphoma,15 and schizophrenia.16 These methods used a combination of systematic literature reviews, experts’ interviews, and data analyses (eg, data visualization, subgroup analyses, or regression modeling techniques). However, it is also possible that the effectiveness of drugs differs across health care settings and across countries. Several studies have explored to which extent the “country effect” was taken into account in RCTs, or which methods could be useful to explore this effect.17,18 Taking account of a between-setting or a between-country difference in drugs’ effectiveness raises numerous methodological questions. For instance, how should a health care setting or a country be characterized: as a unit, or as a set of explicit characteristics? Or what in the “country” may be the source of the difference – if any?
The current case study explored the methods to identify the effect modifiers, at a health care system level. Our objectives were to identify health care system characteristics potentially modifying the effect of antipsychotic drugs and to explore the methods to analyze this modification of effect.
Patients and methods
The study was performed using observational data, to identify the effect modifiers of antipsychotic drugs, at a health care system level. While exploring methods to achieve this goal, the objective of this study was neither to measure the relative effectiveness of antipsychotic drugs nor to generate results to be used outside the context of the current case study.
The European Schizophrenia Outpatient Health Outcome (SOHO) observational cohort dataset was made available to the authors for the purpose of this study. Data were de-identified, and no additional ethics review board or ethics approval was required for the conduct of the current study.
The rationale for using the SOHO database was that this is a wide cohort including observational information on physicians, patients and their condition, antipsychotic drugs prescribed, and patients’ outcome. In short, the SOHO study involved 1100 psychiatrists from across 10 European countries (Germany, Denmark, France, Greece, Ireland, Italy, the Netherlands, Portugal, Spain, and the UK), who recruited 10,218 outpatients with schizophrenia between 2001 and 2002. Participants were at least 18 years of age, required initiation or switch of antipsychotic drugs as per psychiatrists’ clinical judgment, and agreed to participate in the study. All data were collected prospectively by psychiatrists at baseline, 3 months, 6 months, and then every 6 months for 36 months. Further details about the methodologies used can be found elsewhere.19–21 For the current study, only baseline and 3-month data were used.
Data were hierarchical: in each country, psychiatrists were required to recruit 10 patients each on average. We defined the “health care system” using the perspective of a patient who is cared for: a) by one psychiatrist (or the health care team in general), b) in a given health care setting (urban/rural, hospital/outpatient, public/private, etc), and c) in a given country where specific health care policies are applied (reimbursement policies, social security, availability in mental health care, etc). The sources of variability in care care (a, b, and c) were regrouped within the so-called center level, as opposed to the variability between patients (patient level). Data were categorized using the two levels of information: 1) “patient-level” data and 2) “center-level” data.
We hypothesized that patients taken care of in the same “center” (ie, same country and by the same psychiatrist) would form a “cluster,” meaning that they would tend to be similar in terms of treatment patterns and symptom evolution.
Variables and measures
Exposure to antipsychotic drugs
Antipsychotic drugs initiated at baseline were informed by physicians: 52.7% were prescribed olanzapine, 32.9% were prescribed another oral second-generation antipsychotic drug, and 6.7% of patients were prescribed a first-generation oral antipsychotic drug; the remaining patients were prescribed depot drug or a combination of different antipsychotic drugs.19,20
For the current study, exposure to antipsychotic drugs was defined as the initiation of (or switch to) an antipsychotic drug at baseline, regardless of how it is actually used afterward.
The brand/generic names of antipsychotic drugs were de-identified to preserve the noncompetitiveness of the GetReal project and replaced by letters (drug A, drug B, etc.). Patients initiating olanzapine at baseline were oversampled as per the study protocol. We excluded these patients from our analyses, first because many psychiatrists recruited only one patient initiating olanzapine (in this case, heterogeneity of patients’ characteristics was set at null for those psychiatrists). Moreover, we suspected that the choice of olanzapine prescription was driven by the study protocol and not by psychiatrists’ preferences. Including patients on olanzapine would have biased the results on a “cluster effect.” After excluding patients initiating olanzapine, our study population consisted of 4627 patients recruited by 899 psychiatrists.
Antipsychotic drugs initiated at baseline were divided into 1) the most frequently initiated drug (hereafter called “drug A”) and 2) “other” antipsychotic drugs (all other antipsychotic drugs – regrouped), so that groups of approximately equal size could be achieved. Exposure to antipsychotic drugs was dichotomous (drug A vs other).
The severity of schizophrenia symptoms was measured by psychiatrists at baseline and 3 months using the Clinical Global Impression-Severity (CGI-S) scale,22 a single-item scale that rates global severity of symptoms between 1 (lowest severity) and 7 (highest severity).
For the current study, the outcome of interest for patients was the evolution of schizophrenia symptoms 3 months after the initiation of antipsychotic drugs (baseline), computed as the difference between CGI-S scores at 3 months and baseline (∆CGI-S, continuous variable).
All patient-level variables were considered a priori as potential confounders in any observed association between exposure to antipsychotic drugs and patients’ outcome. These variables included the following: patient’s gender, age, housing (independent or not), employment status, duration of illness (time elapsing since the first schizophrenia treatment), substance use disorder (SUD; no structured diagnosis tool was used to diagnose this disorder), number of psychiatric hospitalizations in the previous 6 months, baseline CGI-S score, and patient’s adherence to drug at 3 months as measured by psychiatrists using a single-item scale with four categories: 1) no antipsychotic drug prescribed in the prior 4 weeks; 2) good adherence: “the patient almost always adhered to antipsychotic drugs”; 3) moderate non-adherence: “the patient adhered about half of the time”; and 4) severe non-adherence: “the patient almost never adhered.”
Potential effect modifiers
All the center-level variables listed in the following paragraphs were considered a priori as potential effect modifiers of antipsychotic drugs.
Data collected on the psychiatrist included the following: psychiatrist’s gender, age, length of practice experience, health care practice setting (urban or rural location, as defined by the psychiatrist), and type of practice (private, public, or combined type of practice).
No information on the health care system implemented in each of the 10 European countries participating in this study was directly available in SOHO, except for the country of recruitment. We hypothesized that the performance of a country health care system could modify the effect of antipsychotic drugs. We searched in the literature and in websites of international or European health organizations (eg, World Health Organization [WHO], Organization for Economic Cooperation and Development [OECD], and the European Commission) to identify potentially relevant performance indicators of 1) the health care system in general (ie, not restricted to mental health care) and 2) the mental health care and with data available for the ten countries of interest in SOHO. Two performance indicators were identified and used. First, the Healthcare System Efficiency (HSE) score23 was used as a proxy for the global efficiency of the health care system. This indicator was derived from WHO data on life expectancy at birth (weighted 60%), relative (weighted 30%), and absolute (weighted 10%) cost of health care per capita; higher scores indicated a greater efficiency of the health care system. Second, the number of psychiatrists per 100,000 people, practicing in public or private settings as of 2011 (hereafter called “psychiatrist density”), was used as a proxy for the accessibility to mental health care.24 This indicator was identified in a report from the OECD.25 No other characteristic of countries was used; in particular, the reimbursement policy of each country was not considered because reimbursement policies are rather similar across western European countries. Table 1 summarizes all the variables used in the current study.
Measure of effectiveness
A “core model” for relative effectiveness was built to explore effect modification: patients’ outcome (continuous, normally distributed variable) was computed as a linear function of antipsychotic drug initiation. Using this measure of relative effectiveness, we explored whether center-level variables acted as effect modifiers in the association between antipsychotic drugs and patients’ outcome.
To take account of a cluster effect at a center level, a mixed linear regression model was used, providing an estimate of the association between exposure to antipsychotic drugs and patients’ outcome, with a random intercept being the psychiatrist identifying number.28 The use of a mixed linear regression model provided the opportunity to 1) adjust for potential confounders (as in fixed-effect linear models)29 and 2) measure the relatedness of clustered data using the intra-cluster correlation coefficient (ICC) computed as the ratio between the within-cluster variance and the total variance (within clusters + between clusters).30 Practically, the ICC is the estimation of the unmeasured variance at a psychiatrist level.
First, a multivariable mixed linear regression model was built to take account of patient-level potential confounders (as mentioned earlier) and also to adjust center-level variables (all variables listed earlier, except for psychiatrist’s age which was collinear with his/her length of practice experience). In this model, we identified which of the center-level characteristics were associated with patients’ outcome using a level of significance set at α=0.05.
Second, center-level characteristics found to be associated with patients’ outcome were tested against effect modification. For each center-level characteristic associated with patients’ outcome, the multivariable mixed linear regression model was refitted with the addition of an interaction term between the center-level characteristic and the antipsychotic drug. One model was fitted for each center-level characteristic found associated with patients’ outcome, and, virtually, a maximum of six models were to be built. The center-level characteristic was considered to be an effect modifier if the test on the interaction term returned a significant result (α=0.05).
Finally, in case of statistically significant interaction, the model was stratified by two levels of center-related characteristic, to explore how the strength of association between antipsychotic drug initiation and patients’ outcome was modified.
The CGI-S score at baseline or 3 months was missing for 349 patients, who were therefore excluded, leaving 4278 (92.5%) patients for the analyses: at baseline, 1915 (44.8%) initiated drug A, while 2363 (55.2%) initiated “other” antipsychotic drugs (Figure 1).
Figure 1 Flowchart of patients included in the analyses.
Abbreviation: SOHO, Schizophrenia Outpatient Health Outcome.
Table 2 summarizes the characteristics of patients across the two therapeutic groups. Table 3 provides the health care system performance indicators of the 10 participating countries as well as the characteristics of the 899 participating psychiatrists by country of origin.
Table 3 Characteristics of the 10 participating health care systems and the 899 psychiatrists involved in the recruitment of 4278 patients in SOHO
Notes: aHSE score estimated from life expectancy and relative and absolute costs of health care per capita – source: Bloomberg Business and Financial Information.23 bExpressed/100,000 people – estimation from 2011 (in Spain, this only includes hospital psychiatrists) – source: OECD.25
Abbreviations: HSE, Healthcare System Efficiency; OECD, Organization for Economic Cooperation and Development; SOHO, Schizophrenia Outpatient Health Outcome; NA, not available.
The results of the mixed multivariable linear regression model estimating the association between health care system characteristics and patients’ outcome are detailed in Table 4. The model explained 18% of the overall variance in patients’ outcome (R2=0.18), and the extent of shared variance among patients managed by the same psychiatrist was 23% (ICC=0.23).
No association was found between patients’ outcome and any of the following: psychiatrist’s gender, health care practice setting, type of practice, HSE score, psychiatrist density, and antipsychotic drug initiated at baseline. Length of practice experience was found to be associated with patients’ outcome, with longer experience being associated with less favorable outcomes.
A significant interaction was found between the “length of practice experience” and the “antipsychotic drug” (β=0.009; 95% CI=0.003, 0.015; p<0.001). The multivariable model was then stratified by the length of practice experience (≤15 vs >15 years). The mean change in CGI-S was ∆CGI-S=-0.72 (SD=0.94) in patients managed by practicing psychiatrists with ≤15 years of experience, with a significant association found between patients’ outcome and drug A (vs other antipsychotic drugs) (β=-0.103; 95% CI=-0.180, -0.025; p=0.01). The mean change in CGI-S was ∆CGI-S=-0.68 (SD=0.97) in patients managed by psychiatrists with ≥15 years of experience, and the effect of drug A vs other antipsychotic drugs was not significantly different from 0 (β=0.028; 95% CI=-0.046, 0.101; p=0.46) in this group.
In the current study, the large multinational and observational SOHO cohort was used to explore the effect modification of antipsychotic drugs, at a health care system level. The “relative effectiveness of drug A” (as compared to any other antipsychotic drug) was computed as a mean to elaborate on potential effect modifiers.
Choice of variables
The patients’ outcome was measured by treating psychiatrists using the CGI-S scale, a tool that is easily implemented in observational studies. Its inter-rater reliability is often considered good provided the physicians are adequately trained.31 Therefore, a classification bias (in this study, also confounding) cannot be excluded, particularly considering that younger psychiatrists might have overestimated the level of symptom improvement in their patients, as compared to more experienced psychiatrists. Moreover, the choice of antipsychotic drugs was left at the discretion of treating psychiatrists, and it is also possible that the modification of effect evidenced in our study reflected the fact that younger psychiatrists overestimated the effect of a switch to drug A, as compared to more experienced psychiatrists.
Regarding the health care system, we hypothesized a priori that patients’ outcome would be different, in different countries characterized by the “performance” of corresponding health care systems. In other words, we made the hypothesis that the performance of the health care system would act as an effect modifier of antipsychotic drugs. The first step in the case study consisted of identifying relevant and reliable “performance” indicators. Two indicators were identified: the density in psychiatrist and the global performance of the health care system. The rationale for using the psychiatrist density was that it could act as a proxy for the accessibility to mental health care.24 Previous studies in Europe and the USA have found an association between lower density of psychiatrists and higher suicide rates in the general population32,33 and in patients with schizophrenia.34 In the current study, no association was found between psychiatrist density across the 10 European countries participating in SOHO and symptom evolution in patients, in the short term. It is possible that no such association exists. It is also possible that the patients present in the database were actively followed-up by the medical staff, hence their recruitment into the cohort study. In this case, accessibility to mental health care was in fact not an issue for patients, and the psychiatrist density was not the relevant proxy to use in this study. For further studies aiming at exploring the impact of accessibility to mental health care at a country level, a possible solution would be to use electronic health care records, hence minimizing the impact of conducting an ad hoc study on patients’ sampling. Regarding the global performance of each country, no association was evidenced between global performance and patients’ outcome, and thus, no test of interaction was performed. Other health care system characteristics could also have been tested, following the results of previous studies which have investigated the association between characteristics of health care system and patients’ outcome. In the UK, higher rates of compulsory admission in mental health were found to be associated with a decrease in hospital bed provision over time35 and burnout of health care teams.36 In addition, higher suicide rates were found to be associated with greater turnover of nonmedical staff.37 In the current study, we did not explore these characteristics because data were lacking. Although only outpatients were recruited in the SOHO cohort, it would have been interesting to describe these aspects of setting of outpatient care (eg, staff turnover, nurse visits, and psychotherapy). For instance, a recent study suggested that following antipsychotic drug switch, schizophrenia symptoms tend to improve to a greater extent in patients who benefit from a psychosocial therapy, compared to patients not benefiting from psychosocial therapy.38
Regarding information on the psychiatrist, all variables were used with no hypothesis made a priori. An association was evidenced between psychiatrists’ length of practice experience and patients’ outcome, suggesting more favorable symptom evolution in patients managed by younger psychiatrists, independent of patient-level confounders. This result is in line with results of studies performed in nonpsychiatric39 and psychiatric settings.40 Choudhry et al39 suggested that physicians who have been in practice for longer possess less factual knowledge and are less likely to adopt new effective therapies or may be less receptive to new recommendations, thus explaining poorer patients’ outcome. Previous work from our study group evidenced that patient-reported satisfaction with care was higher in patients managed by younger psychiatrists, independent of patient-related confounders.41 Although not overlapping, the two findings might indicate that, on average, younger psychiatrists may adopt more up-to-date types of management in patients with schizophrenia (eg, shared decision-making and psychotherapy), owing to the fact that the management of schizophrenia has greatly evolved over the past decade.
First, the use of a large and multinational dataset including several levels of information (information on the health care system of a country, on the care setting for patients, and on the prescriber/psychiatrist) allowed us to explore a “cluster effect” in relation with the psychiatrist recruiting patients. Using a mixed model with a random intercept for psychiatrist identifying number revealed that unmeasured confounding at the psychiatrist level explained nearly one-quarter of the variance found for patients’ outcome. Although we cannot specify what exactly is at stake in the “cluster,” this result emphasizes the importance of the role played by health care centers (eg, the psychiatrist) beyond psychotropic drug prescription.
Second, in what concerns effect modification, our analyses were performed in the realm of linear regression models, and thus, using an additive scale. By using a multiplicative scale (eg, logistic regression), different results may have returned.
Third, although effect (measure) modification and interaction are not superimposable,42 we explored effect modification through stratification after showing statistical interaction (Wald test for interaction) for simplicity reasons. Regarding the length of practice experience, the Wald test for interaction (between the length of practice experience and drug A initiation) was statistically significant. This result cannot be generalized to other settings because the measure of “drug A effectiveness” was only used as a mean to explore effect modification and has no validity in itself. Our case study simply illustrates that not only participants should be representative of real-life patients but also the physicians should be representative of real-life prescribers, to ensure the external validity of study results. In the current study, however, the level of modification was not clinically relevant: the mean level of symptom improvement in patients managed by younger psychiatrists was ∆CGI-S=-0.72 (SD=0.94), with a significant effect of drug A (β=-0.103; p=0.010). In patients managed by older psychiatrists, the mean level of symptom improvement was ∆CGI-S=-0.68 (SD=0.97), with the effect of “drug A” being nonsignificantly different from 0 (β=0.028; p=0.457). This result highlights that statistical significance does not mean clinical relevance. In practice, it is not certain that including a biased sample of psychiatrists – owing to their length of practice experience – would alter the accuracy of antipsychotic drug effect estimates.
It is important for decision makers and clinicians to understand how the results of a trial performed across different health care settings and countries would translate into other health care settings and countries with different health care systems. In routine clinical practice, it is important to emphasize that the clinical outcome in a specific patient does not depend solely on the pharmacological effect of the drug prescribed but also on many other factors, related to the patient and health care setting.
To anticipate and take account of this heterogeneity, clinical trials should include information on the important drug effect modifiers at both patient and health care system levels. However, the current case study illustrates the fact that identifying effect modification at a health care system level is far from being straightforward. Effect modification has to be apprehended in the context of the drug (or class of drugs) being taken into account, the effect being measured (whether relative or comparative effectiveness), the primary outcome (whether continuous or dichotomous outcome), the type of model used, the related assumptions (whether additive or multiplicative risk), and the clinical relevance of findings, to name a few examples. Moreover, the identification of relevant health care system characteristics and their availability in databases are also important issues. Further studies are required to explore other methods and other types of data sources to build on knowledge on the identification of effect modifiers.
The work leading to these results has received support from the Innovative Medicines Initiative Joint Undertaking under grant agreement no (115546), resources of which are composed of financial contribution from the European Union’s Seventh Framework Programme (FP7/2007-2013) and European Federation of Pharmaceutical Industries and Associations companies in kind contribution. The supporters had no role in the design, analysis, interpretation, or publication of this study.
All authors contributed toward data analysis, drafting and critically revising the paper and agree to be accountable for all aspects of the work.
Lucien Abenhaim is the chairman of LASER Analytica, and Clementine Nordon has been employed by LASER Analytica, a research and consultancy company collaborating with virtually all pharmaceutical companies. LASER has no commercial interests in any of the products studied. Members of the LASER network have no interest in a drug or other factors studied. Josef Maria Haro works at the Sant Joan de Déu Health Park in Barcelona, Spain. Mark Belger is an employee of Eli Lilly and Company Limited. Helene Verdoux is a clinician at Bordeaux University Hospital. Constance Battin and Tjeerd van Staa report no conflicts of interest in this work.
Depp C, Lebowitz BD. Clinical trials: bridging the gap between efficacy and effectiveness. Int Rev Psychiatry. 2007;19(5):531–539.
Tiihonen J, Wahlbeck K, Lonnqvist J, et al. Effectiveness of antipsychotic treatments in a nationwide cohort of patients in community care after first hospitalisation due to schizophrenia and schizoaffective disorder: observational follow-up study. BMJ. 2006;333(7561):224.
Pagoto SL, Lemon SC. Efficacy vs effectiveness. JAMA Intern Med. 2013;173(13):1262–1263.
Lehman AF, Thompson JW, Dixon LB, Scott JE. Schizophrenia: treatment outcomes research – editors’ introduction. Schizophr Bull. 1995;21(4):561–566.
Fleischhacker WW, Goodwin GM. Effectiveness as an outcome measure for treatment trials in psychiatry. World Psychiatry. 2009;8(1):23–27.
Eichler HG, Abadie E, Breckenridge A, et al. Bridging the efficacy-effectiveness gap: a regulator’s perspective on addressing variability of drug response. Nat Rev Drug Discov. 2011;10(7):495–506.
Murri MB, Guaglianone A, Bugliani M, et al. Second-generation antipsychotics and neuroleptic malignant syndrome: systematic review and case report analysis. Drugs R D. 2015;15:45–62.
Nordon C, Karcher H, Groenwold RH, et al. The “Efficacy-Effectiveness Gap”: historical background and current conceptualization. Value Health. 2016;19(1):75–81.
Stang A, Jockel KH. Avoidance of representativeness in presence of effect modification. Int J Epidemiol. 2014;43(2):630–631.
Longford NT. Selection bias and treatment heterogeneity in clinical trials. Stat Med. 1999;18(12):1467–1474.
Weiss CO, Boyd CM, Wolff JL, Leff B. Prevalence of diabetes treatment effect modifiers: the external validity of trials to older adults. Aging Clin Exp Res. 2012;24(4):370–376.
Huybrechts KF, Mikkelsen EM, Christensen T, et al. A successful implementation of e-epidemiology: the Danish pregnancy planning study “Snart-Gravid.” Eur J Epidemiol. 2010;25(5):297–304.
Innovative Medicines Initiative [homepage on the Internet]. The GetReal Consortium. 2013. Available from: http://www.imi-getreal.eu/. Accessed October 20, 2017.
Ankarfeldt MZ, Adalsteinsson E, Groenwold RH, Ali MS, Klungel OH. A systematic literature review on the efficacy–effectiveness gap: comparison of randomized controlled trials and observational studies of glucose-lowering drugs. Clin Epidemiol. 2017;9:41–51.
Liu L, Hummel N, Mauer M, Morais E. R. O. A systematic literature review on the drivers of effectiveness and the efficacy-effectiveness gap in hematological malignancies with a focus on Hodgkin’s lymphoma. ISPOR. Vienna, Austria: 2016.
Nordon C, Bovagnet T, Belger M, et al. Trial exclusion criteria and their impact on the estimation of antipsychotic drugs effect: a case study using the SOHO database. Schizophr Res. Epub 2017 Jul 13.
Manca A, Willan AR. “Lost in translation”: accounting for between-country differences in the analysis of multinational cost-effectiveness data. Pharmacoeconomics. 2006;24(11):1101–1119.
Ahmad N, Boutron I, Dechartres A, Durieux P, Ravaud P. Applicability and generalisability of the results of systematic reviews to public health practice and policy: a systematic review. Trials. 2010;11:20.
Haro JM, Edgell ET, Novick D, et al. Effectiveness of antipsychotic treatment for schizophrenia: 6-month results of the Pan-European Schizophrenia Outpatient Health Outcomes (SOHO) study. Acta Psychiatr Scand. 2005;111(3):220–231.
Haro JM, Edgell ET, Frewer P, Alonso J, Jones PB, Group SS. The European Schizophrenia Outpatient Health Outcomes Study: baseline findings across country and treatment. Acta Psychiatr Scand Suppl. 2003;(416):7–15.
Haro JM, Edgell ET, Jones PB, et al. The European Schizophrenia Outpatient Health Outcomes (SOHO) study: rationale, methods and recruitment. Acta Psychiatr Scand. 2003;107(3):222–232.
Guy W. Clinical Global Impression. National Institute for Mental Health; 1976.
Bloomberg Business and Financial Information [webpage on the Internet]. Bloomberg Best (and Worst). 2014. Available from: http://www.bloomberg.com/graphics/best-and-worst/#most-efficient-health-care-2014-countries. Accessed October 20, 2017.
European Public Health [webpage on the Internet]. Health System Performance Assessment 2016. 2017. Available from: www.europeanpublichealth.com/health-systems/health-system-performance-assessment/. Accessed August 8, 2017.
Organisation for Economic Co-operation and Development. Health at a Glance 2013: OECD Indicators. 2013. Available from: https://www.oecd.org/els/health-systems/Health-at-a-Glance-2013.pdf. Accessed July 1, 2016.
GBIF. R: A Language and Environment for Statistical Computing [Computer Program]. Vienna, Austria: GBIF; 2008.
Azur MJ, Stuart EA, Frangakis C, Leaf PJ. Multiple imputation by chained equations: what is it and how does it work? Int J Methods Psychiatr Res. 2011;20(1):40–49.
Diez Roux AV. A glossary for multilevel analysis. J Epidemiol Community Health. 2002;56(8):588–594.
Galbraith S, Daniel JA, Vissel B. A study of clustered data and approaches to its analysis. J Neurosci. 2010;30(32):10601–10608.
Killip S, Mahfoud Z, Pearce K. What is an intracluster correlation coefficient? Crucial concepts for primary care researchers. Ann Fam Med. 2004;2(3):204–208.
Pinna F, Deriu L, Diana E, et al. Clinical Global Impression-severity score as a reliable measure for routine evaluation of remission in schizophrenia and schizoaffective disorders. Ann Gen Psychiatry. 2015;14:6.
Tondo L, Albert MJ, Baldessarini RJ. Suicide rates in relation to health care access in the United States: an ecological study. J Clin Psychiatry. 2006;67(4):517–523.
Sher L. Are suicide rates related to the psychiatrist density? A Cross-National Study. Front Public Health. 2015;3:280.
Mittoux A, Tanghoj P, Moore N. Exploring the potential prognostic effect of various country-specific health service data on all-cause mortality using data from a large prospective study in schizophrenia. Pharmacoepidemiol Drug Saf. 2013;22(9):925–932.
Keown P, Weich S, Bhui KS, Scott J. Association between provision of mental illness beds and rate of involuntary admissions in the NHS in England 1988-2008: ecological study. BMJ. 2011;343:d3736.
Priebe S, Fakhoury W, White I, et al. Characteristics of teams, staff and patients: associations with outcomes of patients in assertive outreach. Br J Psychiatry. 2004;185:306–311.
Kapur N, Ibrahim S, While D, et al. Mental health service changes, organisational factors, and patient suicide in England in 1997-2012: a before-and-after study. Lancet Psychiatry. 2016;3(6):526–534.
Ali A, Carre A, Orri M, et al. The real-life effectiveness of psychosocial therapies on social autonomy in schizophrenia patients: results from a nationwide cohort study in France. Psychiatry Res. 2017;247:97–104.
Choudhry NK, Fletcher RH, Soumerai SB. Systematic review: the relationship between clinical experience and quality of health care. Ann Intern Med. 2005;142(4):260–273.
Lee HC, Lin HC. Are psychiatrist characteristics associated with postdischarge suicide of schizophrenia patients? Schizophr Bull. 2009;35(4):760–765.
Nordon C, Rouillon F, Barry C, Gasquet I, Falissard B. Determinants of treatment satisfaction of schizophrenia patients: results from the ESPASS study. Schizophr Res. 2012;139(1–3):211–217.
VanderWeele TJ. On the distinction between interaction and effect modification. Epidemiology. 2009;20(6):863–871.
This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at https://www.dovepress.com/terms.php and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.Download Article [PDF]