Quality and utilization of the Finnish clinical practice guideline in schizophrenia: evaluation using AGREE II and the vignette approach

Introduction

Clinical practice guidelines are systematically developed statements aiming to assist health care decisions,^1,2 enhance the quality of care, provide aid for recognizing diagnoses, and address issues relating to health, prevention, and treatment of diseases.³ Still, assessment procedures too often only rely on physicians’ opinions.^4,5 At their best, the guidelines can support legislators, decision-makers, health care professionals, patients, and their families in making decisions related to treatment.^6,7 Unfortunately, guidelines are often based on experts’ opinions rather than systematic and methodology-based rigor development processes.⁸ Other problems include unclear links between evidence and recommendations^9,10 and low quality, which may hinder their use in daily practice.^11–16 Stakeholders’ views and editorial independence are also concerns.^14–17

Various approaches have been developed to support the quality of guidelines. They are, for example, Grading of Recommendations Assessment, Development and Evaluation to develop guidelines,¹⁸ standards for guideline development by the Guidelines International Network to minimize the quality differences among guidelines,¹⁹ and the Appraisal of Guidelines for Research and Evaluation (AGREE) to evaluate the methodological quality of guidelines.²⁰ Furthermore, the modified version of AGREE II²¹ has been used for guideline development and evaluation.^{9,10,14,15,17,22–25} Previous studies have used the AGREE instrument in different settings. Sinclair et al¹⁰ used AGREE to assess whether any improvement in the quality of the guideline could be identified in various WHO guidelines. They found that the guidelines (N=20) published since 2007 have become more systematic and transparent.¹⁰ Gaebel et al²⁶ evaluated the methodological quality of schizophrenia guidelines (N=5) and found that the quality of the guidelines related to schizophrenia has improved.²⁶

Although the assessment of the content of the guidelines has shown improvement in scope and purpose of the guidelines,^14,15 there is still room to improve the guideline development process. Particularly in mental health settings, challenges and deficiency in implementation of guidelines have been recognized.^27,28 There are several guideline implementation barriers in mental health care, such as that interventions may be successful in one country but inappropriate in another because of differences in roles of caregivers, other stakeholders, and the legal protections available to workers. The recommendations may also be presented in an unclear and/or ambiguous way. More barriers exist if national guidelines are not accessible via international databases, are not available in English, or do not adopt a common structure.¹⁴ Guideline implementation barriers, especially regarding schizophrenia, can arise if patients and/or their families do not question clinicians about the basis of the care and the reasoning behind decisions, ie, clinicians are not encouraged to think about their reasoning, and the need for guidelines may therefore not be seen as acute. Moreover, there should not be an imbalance between what is recommended and what is actually done, and therefore, auditing and feedback of patient outcomes are essential for providing professionals and decision makers with information about the true impact of the practice.¹⁶

Feedback from health care professionals is often neglected as well.²⁹ To ensure the implementation of guidelines in daily practice,^30–32 the quality of the guidelines should be high.^{10,23,28,33,34} On the other hand, it has already been noticed that some professional groups do not use treatment guidelines.¹⁶ We therefore conducted a study where team members from various professional backgrounds evaluated the Finnish clinical practice guideline to support treatment of persons with schizophrenia (the Current Care Schizophrenia Guideline).³⁵ We also evaluated how team members were able to utilize the guideline in a hypothetical case in psychiatric services.

Participants and methods

Design

A descriptive quantitative study design was used to evaluate the Finnish Current Care Schizophrenia Guideline (hereafter the Schizophrenia Guideline),³⁵ from the perspective of its quality and utilization.

Guideline

The Schizophrenia Guideline³⁵ was evaluated in this study. The guideline, based on national and international knowledge, excluding childhood schizophrenia, was developed in 2001 and updated in 2008, 2013, and 2015 to be used in Finland. The guideline was selected because schizophrenia is a long-term and chronic condition,³⁶ and is one of the costliest illnesses for patient populations.^37,38 The guideline provides a framework for the content of treatment and service organizations taking care of patients with schizophrenia.^35,39 The Schizophrenia Guideline includes 27 pages in Portable Document Format (PDF). The guideline consists of 22 headings and 37 sub-headings, such as target group, epidemiology, a model of illness and prognosis, prevention, risk factors, preliminary symptoms of psychosis, identification of first psychosis and its treatment, examination of the disease and diagnosis, treatment and rehabilitation, medication, psychosocial treatment methods including education, work, housing, and other variables in treatment, cooperation between organizations and mental health services, and an economic perspective. The principles for treatment at different phases of the illness are divided into four parts as follows: acute, balanced period, stable period, and prevention of relapses. References are listed at the end of the guideline. In addition, in the beginning of the guideline a summary is provided. From this summary, eight key recommendations can be identified (Outpatient care, Treatment relationship, Care plan, Medication, Psychosocial individual care, Family psychoeducation, Psychosocial rehabilitation, and Occupational rehabilitation). These eight key recommendations were utilized in the present study while analyzing the vignette responses of the participants.

Settings

The data were collected from two cities in southern Finland. Both cities offered specialized psychiatric services with one inpatient psychiatric ward in one city and seven outpatient units in community services in the other city (Mental Health Act, 1116/1990,⁴⁰ Health Care Act 1326/2010,⁴¹ and Ministry of Social Affairs and Health 2018⁴²); this ward and these outpatient units were used as data collection points. Observing different types of services allows us to capture a wide range of views of professionals.⁴³ The inpatient psychiatric ward and individual outpatient units were selected to focus on patients with schizophrenia or other psychotic disorders.⁴⁴

The inpatient psychiatric ward included 12 hospital beds. The majority of patients admitted to the ward had dual diagnoses with an average length of stay of about 2 weeks. In the outpatient care services, the seven units included three psychiatric units (open 8 hours/day, 5 days/week), one acute psychiatric clinic (open 12 hours/day, 7 days/week), one psychiatric mobile team visiting patients at home (open 12 hours/day, 7 days/week), and two psychiatric day hospital wards (open 8 hours/day, 5 days/week). In the outpatient units, the length of the treatment ranged between one acute visit to a 3-year rehabilitation process with several visits.

Study population and recruitment

The population of the study included a team of professionals (N=67); 21 worked in the hospital ward and 46 in outpatient care units. Everyone (total sample) working in these settings were invited to participate in the study (doctors, registered nurses, practical nurses, psychologists, social workers, and assistant personnel). The recruitment was based on willingness to participate in the study. Possible participants received information about the study both orally and in written format. If willing to participate, they signed the written informed consent form. Out of the 67 possible participants, 21 staff members in the psychiatric ward and 28 in the outpatient units participated in the study (n=49, response rate 73%).

Study instruments

The AGREE II

The AGREE II instrument²¹ was used to evaluate the methodological quality of the guideline. The AGREE instrument was translated into Finnish using a double-translation process, following the instructions of the AGREE II Translation Protocol.^45,46 The instrument was translated in Finnish by four independent researchers, and consensus for the translation was reached within a group. Then, two independent translators back-translated the instrument in English, and the back-translations were compared by the AGREE to the original document. The AGREE pointed out the terms that needed more work. These terms were rechecked and sent to the translator again in Finland, and the final approval for the translation was received from the AGREE.

The instrument includes 23 items, which are divided into six domains as follows: scope and purpose (three items), stakeholder involvement (three items), rigor of development (eight items), clarity of presentation (four items), applicability (three items), and editorial independence (two items). Participants were asked to rate each item using a seven-point scale (7 = strongly agree, 1 = strongly disagree). Out of the six domains, the highest scores indicate the high-quality domain (domain score ≥ 70%).⁴⁷ Detailed domains and definitions of items are described in Table 1.

Table 1 Main definition of domains in the AGREE II instrument Abbreviation: AGREE II, Appraisal of Guidelines for Research and Evaluation II.

The participants were also asked to evaluate whether the items were well written, if the descriptions were clear and succinct, and whether the content of each domain could easily be found in the guideline. They were further instructed to write down if they thought any information was still missing and why they may have thought this. In addition, the participants evaluated the overall quality of the guideline (7 = the highest possible quality, 1 = the lowest possible quality) and rated whether they would recommend the guideline for use in clinical practice (Yes – Yes, with modifications – No). In all items, a higher percentage score indicated a higher quality of each domain.

Vignette

A vignette framework, adopted from Zadvinskis and Grudell,⁴⁸ was used to evaluate the utilization of the Schizophrenia Guideline.³⁵ A vignette provided the opportunity to gain valuable in-depth insight into how participants approached the guideline.⁴⁹ The participants were instructed to read the vignette about a young adult and his family members who had questions related to his treatment of schizophrenia. The respondents were then instructed to find out answers to the following questions using the guideline as a source of reference: 1) Where would the treatment take place? 2) What would the treatment include? The vignette was as follows:

When Kalle was 20 years old, his behaviour started to change. He alienated himself from the others and heard threatening voices. Eventually, voices and fears related to them got worse, and Kalle no longer understood what was real and what was not. At last, Kalle’s parents took him to the health care centre. Kalle received a referral to a psychiatric hospital, where he was treated for two months. At the age of 21, Kalle was diagnosed with paranoid schizophrenia. Kalle’s family was shocked at the diagnosis. They had no prior knowledge or experience with schizophrenia, and the illness was not familiar to them. The family wanted to support Kalle, and they especially requested information on the treatment of schizophrenia. They were particularly interested in what would happen when Kalle was discharged from the hospital. Where would the treatment take place, and what would it include? Your next task is to seek answers to the questions introduced by Kalle’s family from the Current Care Guideline for Schizophrenia.

Data collection

The data were collected between May and September 2016. The data collection process included the following two parts: an evaluation of the quality of the Schizophrenia Guideline using the AGREE II instrument and utilization of the guideline by means of a vignette approach. The data were collected in seven small groups, with a total of 49 participants. Each data collection session took about 90 minutes. The length of the sessions was set according to the AGREE II instructions²¹ and previous experience by Dans and Dans.⁵⁰

First, the participants received the Schizophrenia Guideline, orientated themselves with its content, and wrote answers to the vignette questions. Second, they received the AGREE II instrument²¹ to evaluate the guideline.¹⁰ Third, to improve the standardization of the data collection and ensure that all participants had an adequate understanding of the instrument contents,^24,51 the researcher summarized aloud the content of each of the domains and items and instructed the response procedure. Finally, the participants filled in the AGREE II instrument independently, without discussing with other participants.²⁴

Ethics statement

The study procedure was approved by the ethics committee of the Hospital District of Southwest Finland (ETMK: 9/1801/2016). The permission to conduct the study was obtained from the involved health care service organizations (decision dates: April 4, 2016:49 and June 16, 2016). The written informed consent form was signed by every participant.

Analysis

The quality score of each domain (eg, scope and purpose and stakeholder involvement) of the AGREE II instrument was calculated by summing all the scores of the individual items and by scaling the total score as a percentage. To standardize each domain, the percentage score was calculated as follows: (obtained score — minimum possible score)/(maximum possible score — minimum possible score) × 100; the higher score suggested the higher quality of each domain, 100% being the maximum.²¹ Descriptive and statistical data (%, mean, minimum, maximum) were analyzed using IBM SPSS Statistics version 24 for Windows (© Copyright IBM Corporation 1994, 2016). The results were reported using The AGREE Reporting Checklist and percentage of each domain.^52,53

In the analysis of the utilization of the vignette, two researchers (VP and TL) independently rated whether the eight key recommendations (Outpatient care, Treatment relationship, Care plan, Medication, Psychosocial individual care, Family psychoeducation, Psychosocial rehabilitation, and Occupational rehabilitation) of the Schizophrenia Guideline (Table 2) were deductively found in the respondents’ answers. In this task, a modified method of Münzberg et al⁵⁴ was used. If the participants indicated that Kalle’s treatment was in line with the eight key recommendations, the rating value was 1 (yes = 1); if not, the value was 0. The participants’ “yes” answers were then added together for a score (range 0–8); the higher the score, the better the respondents’ answers represented the content of the guideline.

Table 2 Evaluation of the utilization of the Schizophrenia Guideline with a vignette Note: Schizophrenia Guideline, Finnish Current Care Schizophrenia Guideline.

The results were then recategorized.⁵⁴ If all eight key recommendations were recognized in a respondent’s written answers, it was categorized as “In agreement”. If five to seven key recommendations were detected in the written text, the result was categorized as “Minor variations”. Furthermore, if one to four key recommendations were found, the result was rated as “Major variations”. Finally, if none of the key recommendations were found in a respondent’s answers, the result was evaluated as “no statement”. After that, the participants’ end results (in agreement–minor variation–major variation–no statement) were totaled and a conformity percentage was calculated. This was done to see, for example, how many participants reached full “in agreement” in their answers. Moreover, for each individual key recommendation, it was aggregated how many of the participants had written down that particular recommendation out of the eight.

Results

Characteristics of the respondents

A total of 49 respondents described the quality of the Current Care Guideline for Schizophrenia. Respondents were from two cities (21 and 28 respondents). Their professions were as follows: doctors (n=2), registered nurses (n=39), practical nurses (n=2), a psychologist (n=1), a social worker (n=1), assistant personnel (n=2), and not known (n=2).

Quality of the guideline based on AGREE II

Most of the respondents agreed (77%) that the overall objectives of the guideline, relating to health questions and the target population, were found in the guideline. On the contrary, it was unclear whether the founding body and competing interests were clearly indicated in the guideline (42%) (Table 3).

Table 3 AGREE II domains and quality evaluation of the Current Care Schizophrenia Guideline Abbreviation: AGREE II, Appraisal of Guidelines for Research and Evaluation II.

The overall mean score evaluation for the quality of the Schizophrenia Guideline was 73% (out of a maximum of 100%). The mean score in item overall quality was five (out of seven), the minimum was three and the maximum was seven, according to the responses of 45 appraisers. Use of the guideline would be recommended by 57% of the participants (n=28). Over one-third of the participants (n=17, 37%) would recommend guideline use with modification. The recommended modifications were, for example, clarification of presentation to patients and families, such as clearer summaries, tables, and materials.

Guideline utilization by means of a vignette approach

Participants produced answers to the vignette questions about the treatment of schizophrenia according to the Finnish Current Care Schizophrenia Guideline. The answers were recategorized to the eight key recommendations of the guideline. Most of the answers lacked a few or more recommendations: 80% of the end results had either “minor” or “major” variations. Overall, an 18% conformity rate was established between the guideline’s key recommendations and the participants’ answers, ie, nine out of the 49 respondents reached the end result “in agreement” (Table 4).

Table 4 The results after recategorizing the vignette answers Notes: A number of recommendations recognized.61 Scoring range: 8 = “In agreement”; 5–7 = “Minor variations”; 1–4 = “Major variations”; and 0 = “No statement”.

The most commonly mentioned key recommendation was “antipsychotic medication” (n=41, 84%), followed by “psychosocial rehabilitation” (n=40, 82%) and “outpatient care” (n=39, 80%). The least mentioned key recommendations were “long-term, trustful treatment relationship” (n=19, 39%) and “occupational rehabilitation and supported employment” (n=19, 39%) (Table 5).

Table 5 The guideline’s key recommendations and the participants’ answers

Discussion

In this study, we assessed the quality of guideline for schizophrenia using AGREE II. We found that, out of the six domains, the overall objective of the guideline, health questions, and target population (domain “scope and purpose”) received the highest quality scores. These results were similar to those of previous investigations.^23,24 Based on a previous study, the domain “rigor of development”, including methodology for searching and selecting evidence, increased the validity of the guideline.²³ In this study, the mean score of “rigor of development” was 56%, which shows that the description of methodology was comprehensive. The overall evaluation of the quality was also quite high (73%). Over half (57%) of the participants reported that they would recommend the guideline, and over one-third (37%) answered that they would recommend it with modifications. As in previous studies, the domain “clarity of presentation” received a high score, which supports the notion that the recommendations are clear and easy to identify.^10,15,17 Our result is promising, as unclear methodology and content may form a barrier, preventing a guideline from being utilized in practice.^26,55 We can therefore assume that this guideline for schizophrenia is easy to implement.

On the other hand, “applicability” of the guideline was assessed relatively poorly. In this context, “applicability” referred to facilitators and barriers to application, advice or tool, resources for implementation, and monitoring and/or auditing criteria. Based on previous studies, the barriers for implementation of guidelines may be organizational barriers,¹² a lack of knowledge or motivation among staff members to use guidelines, negative attitudes or disagreement among staff or service users toward a guideline, or concern about financial motives behind guideline recommendations.^6,12,56 Also, the lack of trust of the sources or researchers behind the studies may be a barrier for implementing the guideline.⁵⁷ Nurses have also claimed that they do not have enough time or resources to follow the guidelines.⁵⁸ Our finding is important because facilitators and barriers to guideline application are important elements when guidelines are implemented into clinical practice.^23,24,50

To facilitate guideline application, health care professionals and other experts should actively participate in the guideline development.^19,58 The guideline development process for the Finnish clinical practice guideline for schizophrenia did not include nurses or multidisciplinary teams. Representatives of patients and their families were also missing in the guideline development process, which was reflected by a low score in “stakeholder involvement”, as in the previous study by Jin et al.²⁴ Indeed, in this study, guideline utilization in the vignette exercise revealed that most of the participants were unable to find some of the eight key guideline recommendations. This may indicate that participants were not familiar with the guideline beforehand, although, as professionals working with patients with schizophrenia, they should have been. Furthermore, we may wonder how this exercise reflects clinical reality. If the professionals in psychiatric practice do not recognize the importance of, eg, psychosocial interventions in the care of patients with schizophrenia, there is an urgent need to assure care quality and provision.

Good results for the implementation of the clinical guidelines have been described when implementation approach has been comprehensive at different levels.^11,59,60 Positive outcomes have also been achieved when health professional teams have been closely connected with the facilitators of the implementation process,¹² the specific development framework for implementation,²⁸ or multifaceted interventions,³² as well as when there has been effective leadership.⁶¹ Any staff expected to put the guideline into practice should be fully aware of the content of the guideline, especially doctors.⁶²

Several strategies have been proposed to promote guideline implementation, such as using a practical checklist with interventions linked to a specific care sequence, providing teaching sessions, and sharing successful implementation strategies.^6,63 Moreover, guideline implementation has been promoted with the introduction of a system consultant approach,⁶⁴ to accelerate the rate at which a guideline is put into practice, a training protocol,⁶⁵ to improve the efficiency of guideline implementation, and the Study of Practices Enabling Implementation and Adaptation in the Safety Net,⁶⁶ and to compare implementation strategies that have been introduced. In addition to these, summary documents or quick reference guides are also needed. In this guideline for schizophrenia, facilitators and barriers to its application as well as its implementation were described as poor. However, additional online material is available to support the education about diagnosis settings and regarding treatment and rehabilitation.³⁵

Strengths and limitations

The participants were staff members working in psychiatric services, which gave them a starting point to assess the guideline based on their own experiences. Their work with patients suffering from schizophrenia certainly provided them with insight for the evaluation task. The number of evaluators in our study was less but still more than those of previous studies,^{9,14,15,21,24,50} which is a quality of this study that supports the representativeness of the results in daily practice. Having two study sites, the hospital ward and the outpatient units, also allowed for a wider evaluation approach for the study.²⁸ Still, generalization of the findings is difficult. The validity of evaluation depends on the raters. In this study, nurses formed the most central group. This reflects the reality of clinical practice, where nurses spend most of the time in direct patient care, compared to other professional groups, eg, medical doctors.^67,68 Furthermore, work in mental health care is typically done by multiprofessional teams, and guidelines are implemented as teams.¹² In a systematic review by Bekkering et al,⁹ a large variation existed among professionals at whom the guidelines were targeted. Therefore, we consider the composition of our sample to be justifiable.

The study has some limitations that should be noted. The evaluators commented that the time given to find out significant facts for the vignette⁴⁹ and answer all 23 items of the AGREE II instrument was very less. The time allotted for the evaluation of the guideline using the AGREE II instrument was based on previous recommendations.^21,50 According to the AGREE II Instrument, the information in the guideline should be found easily, and therefore a lack of the appropriate amount of time to evaluate the guideline may tell us that the guideline is too complicated and not easy to use. Grimmer et al⁶⁹ found AGREE II to be complex, compared with a rapid instrument International Centre for Allied Health Evidence (iCAHE), when evaluating clinical practice guidelines. We may therefore question if we have indeed used the best tool for the guideline evaluation. In addition, we used paper format (PDF) only. In the instructions of AGREE II, it is stated that evaluators should first carefully familiarize themselves with all the additional materials, including any public websites available.²¹ In our evaluation process, access to websites was not available, and therefore, the evaluators did not benefit from all available materials. For example, in the Schizophrenia Guideline, in addition to the 352 references in the PDF format, the available list of review references includes 63 online references. A lack of access to these references may have decreased the capacity to answer all the questions. On the other hand, if online access would have been available, the evaluators may not have had time to read all the resources in the given time period. In turn, the AGREE II instrument is based on the fact that the item contents should be easy to find in the guideline.²¹ Additionally in our study, the guideline evaluation was conducted by groups including only professionals, which is a limitation of our study. Including user groups (service users and their families) would have brought depth to the evaluation of the guideline. This, however, reflects to some extent the situation in Finnish psychiatric care; user involvement, eg, in service development, might not be a standard practice, and especially in inpatient care, professionals may have a negative attitude toward developing services in cooperation with patients.⁷⁰ Finally, we used only one vignette in our study to evaluate utilization of the guideline. Using multiple vignettes may have made it possible to obtain a more comprehensive evaluation. However, a single-vignette approach combined with the AGREE appraisal exercise has been used previously by other researchers as well,⁴⁸ and we may argue that using a short single vignette reflects clinical reality, where clinicians may have limited information about a patient’s situation when giving advice, eg, to families. On the other hand, according to a systematic study about vignettes,⁷¹ respondents are unable to integrate and process large information quantities that are provided simultaneously, or they may lose attention when sifting through too many vignettes. In addition, we only analyzed the key recommendations of the guideline; thus, it was achievable for the raters to detect all of them while utilizing the guideline with the vignette.

Conclusion

Overall, the quality of the Schizophrenia Guideline, evaluated based on the AGREE II instrument, was generally high. Moreover, the utilization of the guideline seems to be good, and it was recommended by over 90% with some modifications. On the other hand, not all staff members were able to find all the key recommendations presented in the guideline. Still, on the whole, the Schizophrenia Guideline was evaluated to be an appropriate tool for use in clinical practice. Therefore, its use can be recommended in future schizophrenia treatment development.

This study provides insight into the quality and utilization of the Finnish Current Care Schizophrenia Guideline. The guideline was evaluated to be an appropriate tool for clinical practice according to the team members from various professional backgrounds in the following two care settings: a psychiatric ward and outpatient units. By emphasizing the weaker aspects of the guideline, such as stakeholder involvement, its quality could be improved upon and then become implemented more into daily practice. We hope this study is found to be valuable to professionals working in psychiatry and all the other health care settings, regarding how guidelines and the transparency of their content can be evaluated. Fundamentally, the ability of patients to receive care based on best practices should be the goal of the health care sector. Therefore, it is each health care professional’s duty to be familiar with and use the relevant guidelines in his/her practice. Our results reveal that this standard may not be being met. Professionals’ competence and knowledge levels about using clinical practice guidelines merit more attention.

Acknowledgment

The authors would like to thank Leigh Ann Lindholm, MA, for language editing. The study was funded by the Academy of Finland (projects 294298, 307367), Hospital District of Southwest Finland (13893) and the University of Turku.

Disclosure

The authors report no conflicts of interest in this work.

---

References

1.		Field MJ, Lohr KN, eds. Clinical Practice Guidelines: Directions for a New Program. Institute of Medicine (US) Committee to Advise the Public Health Service on Clinical Practice Guidelines. Washington (DC): National Academies Press (US); 1990.
2.		World Health Organization. Global Programme on Evidence for Health Policy. Guidelines for Who Guideline. Geneva: World Health Organization; 2003. Available from: http://apps.who.int/iris/bitstream/handle/10665/68925/EIP_GPE_EQC_2003_1.pdf;jsessionid=62D743B320A41FE0D355218F764D5B2D?sequence=1. Accessed October 17, 2018.
3.		Graham R, Mancher M, Miller Wolman D, et al., eds. Clinical Practice Guidelines We Can Trust. Committee on Standards for Developing Trustworthy Clinical Practice Guidelines, Institute of Medicine. Washington (DC): National Academies Press; 2011.
4.		Kuronen R, Jallinoja P, Patja K. Use of and attitudes toward current care guidelines among primary and secondary care nurses in Finland. Clin Nurs Res. 2011;20(3):310–325. doi:10.1177/1054773811407765
5.		Kim J, Bates DW. Medication administration errors by nurses: adherence to guidelines. J Clin Nurs. 2013;22(3–4):590–598. doi:10.1111/j.1365-2702.2012.04344.x
6.		Yang J, Han C, Yoon HK, et al. Experiences and barriers to implementation of clinical practice guideline for depression in Korea. BMC Psychiatry. 2013;13:150. doi:10.1186/1471-244X-13-150
7.		Francke AL, Smit MC, de Veer AJ, et al. Factors influencing the implementation of clinical guidelines for health care professionals: a systematic meta-review. BMC Med Inform Decis Mak. 2008;8:38. doi:10.1186/1472-6947-8-38
8.		Oxman AD, Lavis JN, Fretheim A. Use of evidence in WHO recommendations. Lancet. 2007;369(9576):1883–1889. doi:10.1016/S0140-6736(07)60675-8
9.		Bekkering GE, Aertgeerts B, Asueta-Lorente J-F, et al. Practitioner review: evidence-based practice guidelines on alcohol and drug misuse among adolescents: a systematic review. J Child Psychol Psychiatry. 2014;55(1):3–21. doi:10.1111/jcpp.12145
10.		Sinclair D, Isba R, Kredo T, et al. World Health Organization guideline development: an evaluation. PLoS One. 2013;8:e63715. doi:10.1371/journal.pone.0063715
11.		Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients’ care. Lancet. 2003;362(9391):1225–1230. doi:10.1016/S0140-6736(03)14546-1
12.		Forsner T, Hansson J, Brommels M, et al. Implementing clinical guidelines in psychiatry: a qualitative study of perceived facilitators and barriers. BMC Psychiatry. 2010;10:8. doi:10.1186/1471-244X-10-8
13.		Van Duin D, Franx G, Van Wijngaarden B, et al. Bridging the science-to-service gap in schizophrenia care in the Netherlands: the Schizophrenia Quality Improvement Collaborative. Int J Qual Health Care. 2013;25(6):626–632. doi:10.1093/intqhc/mzt072
14.		Joosen MCW, Brouwers EBM, van Beurden KM, et al. An international comparison of occupational health guidelines for the management of mental disorders and stress-related psychological symptoms. Occup Environ Med. 2015;72(5):313–322. doi:10.1136/oemed-2013-101626
15.		Sabharwal S, Patel NK, Gauher S, et al. High methodologic quality but poor applicability: assessment of the AAOS guidelines using the AGREE II instrument. Clin Orthop Relat Res. 2014;472:1982–1988. doi:10.1007/s11999-014-3530-0
16.		Bighelli I, Ostuzzi G, Girlanda F, et al. Implementation of treatment guidelines for specialist mental health care. Cochrane Database Syst Rev. 2016;15:CD009780.
17.		Polus S, Lerberg P, Vogel J, et al. Appraisal of WHO guidelines in maternal health using the AGREE II assessment tool. PLoS One. 2012;7:e38891. doi:10.1371/journal.pone.0038891
18.		Austin TM, Richter RR, Sebelski CA. Introduction to the GRADE approach for guideline development: considerations for physical therapist practice. Phys Ther. 2014;94(11):1652–1659. doi:10.2522/ptj.20130627
19.		Qaseem A, Forland F, Macbeth F, et al. Guidelines International Network: toward international standards for clinical practice guidelines. Ann Intern Med. 2012;156(7):525–531. doi:10.7326/0003-4819-156-7-201204030-00009
20.		The AGREE Collaboration. Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care. 2003;12(1):18–23.
21.		Brouwers M, Kho ME, Browman GP, et al. AGREE II: advancing guideline development, reporting and evaluation in healthcare. CMAJ. 2010;182(18);E839–E842. doi:10.1503/cmaj.092053
22.		Makarski J, Brouwers MC, AGREE Enterprise. The AGREE Enterprise: a decade of advancing clinical practice guidelines. Implement Sci. 2014;9:103. doi:10.1186/s13012-014-0103-2
23.		Armstrong JJ, Goldfarb AM, Instrum RS, et al. Improvement evident, but still necessary in clinical practice guideline quality: a systematic review. J Clin Epidemiol. 2016;81:13–21. doi:10.1016/j.jclinepi.2016.08.005
24.		Jin Y, Wang Y, Zhang Y, et al. Nursing practice guidelines in China do need reform: a critical appraisal using the AGREE II instrument. Worldviews Evid Based Nurs. 2016;13(2):124–138. doi:10.1111/wvn.12127
25.		Bennett K, Gorman DA, Duda S, et al. Practitioner review: on the trustworthiness of clinical practice guidelines – a systematic review of the quality of methods used to develop guidelines in child and youth mental health. J Child Psychol Psychiatry. 2016;57(6):662–673. doi:10.1111/jcpp.12547
26.		Gaebel W, Riesbeck M, Wobrock T. Schizophrenia guidelines across the world: a selective review and comparison. Int Rev Psychiatry. 2011;23(4):379–387. doi:10.3109/09540261.2011.606801
27.		Girlanda F, Fiedler I, Ay E, et al. Guideline implementation strategies for specialist mental healthcare. Curr Opin Psychiatry. 2013;26:369–375. doi:10.1097/YCO.0b013e328361e7ae
28.		Fiscler I, Riahi S, Stuckey MI, et al. Implementation of a clinical practice guideline for schizophrenia in a specialist mental health center: an observational study. BMC Health Serv Res. 2016;16:372. doi:10.1186/s12913-016-1618-9
29.		Corry M, Clarke M, While AE, et al. Developing complex interventions for nursing: a critical review of key guidelines. J Clin Nurs. 2013;22(17–18):2366–2386. doi:10.1111/jocn.12173
30.		Grimshaw JM, Thomas RE, MacLennan G, et al. Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess. 2004;8(6):1–72. doi:10.3310/hta8060
31.		Weinmann S, Koesters M, Becker T. Effects of implementation of psychiatric guidelines on provider performance and patient outcome: systematic review. Acta Psychiatr Scand. 2007;115(6):420–433. doi:10.1111/j.1600-0447.2007.01016.x
32.		Gagliardi A, Brouwers M, Palda V, et al. How can we improve guideline use? A conceptual framework of implementability. Implement Sci. 2011;6:26. doi:10.1186/1748-5908-6-26
33.		Turner T, Misso M, Harris C, et al. Development of evidence-based clinical practice guidelines (CPGs): comparing approaches. Implement Sci. 2008;3:45. doi:10.1186/1748-5908-3-45
34.		Dizon JM, Machingaidze S, Grimmer K. To adopt, to adapt, or to contextualise? The big question in clinical practice guideline development. BMC Res Notes. 2016;9:442. doi:10.1186/s13104-016-1938-1
35.		Duodecim. Current Care Guideline of Schizophrenia. The Finnish Medical Society Duodecim; 2015. Available from: http://www.kaypahoito.fi/web/english/guidelineabstracts/guideline?id=ccs00038. Accessed October 17, 2018.
36.		An der Heiden W, Häfner H. Investigating the long-term course of schizophrenia by sequence analysis. Psychiatry Res. 2015;228(3):551–559. doi:10.1016/j.psychres.2015.04.056
37.		OECD. Fit Mind, Fit Job: From Evidence to Practice in Mental Health and Work. OECD Publishing; 2015. Available from: http://www.oecd-ilibrary.org/employment/fit-mind-fit-job_9789264228283-en. Accessed October 17, 2018.
38.		Gustavsson A, Svensson M, Jacobi F, et al. Cost of disorders of the brain in Europe 2010. Eur Neuropsychopharmacol. 2011;21(10):718–779. doi:10.1016/j.euroneuro.2011.08.008
39.		Ministry of Social Affairs and Health. National Action Plan to Reduce Health Inequalities 2008–2011. Helsinki: Publications of the Ministry of Social Affairs and Health 25; 2008. Available from: https://julkaisut.valtioneuvosto.fi/bitstream/handle/10024/71185/Julk200825.pdf?sequence=1. Accessed October 17, 2018.
40.		Mental Health Act 1116/1990. Finlex. Available from: http://www.finlex.fi/en/laki/kaannokset/1990/en19901116_20101338.pdf. Accessed October 17, 2018.
41.		Health Care Act 1326/2010. Finlex; 2011. Available from: http://www.finlex.fi/en/laki/kaannokset/2010/en20101326. Accessed October 17, 2018.
42.		Ministry of Social Affairs and Health. Mental health services; 2018. Available from: http://stm.fi/en/mental-health-services. Accessed October 17, 2018.
43.		Kozel B, Grieser M, Abderhalden C, Cutcliffe JR. Inter-rater reliability of the German version of the Nurses’ Global Assessment of Suicide Risk scale. Int J Ment Health Nurs. 2016;25(5):409–417. doi:10.1111/inm.12193
44.		National Institute for Health and Welfare. Psychiatric specialised health care 2014. Statistical report, July 2016. Available from: http://urn.fi/URN:NBN:fi-fe2016051612397. Accessed October 17, 2018.
45.		Råholm MB, Thorkildsen K, Löfmark A. Translation of the Nursing Clinical Facilitators Questionnaire (NCFQ) to Norwegian language. Nurse Educ Pract. 2010;10(4):196–200. doi:10.1016/j.nepr.2009.08.005
46.		The AGREE research trust. Method for translation AGREE II; 2016. Available from: http://www.agreetrust.org/wp-content/uploads/2013/06/Methods_for_translating_the_AGREE_II.pdf. Accessed October 17, 2018.
47.		Duda S, Fahim C, Szatmari P, Bennett K. Is the National Guideline Clearinghouse a trustworthy source of practice guidelines for child and youth anxiety and depression? J Can Acad Child Adolesc Psychiatry. 2017;26(2):86–97.
48.		Zadvinskis IM, Grudell BA. Clinical practice guideline appraisal using the AGREE instrument: renal screening. Clin Nurse Spec. 2010;24(4):209–214. doi:10.1097/NUR.0b013e3181e36072
49.		Jackson M, Harrison P, Swinburn B, Lawrence M. Using a qualitative vignette to explore a complex public health issue. Qual Health Res. 2015;25(10):1395–1409. doi:10.1177/1049732315570119
50.		Dans AL, Dans LF. Appraising a tool for guideline appraisal (the AGREE II instrument). J Clin Epidemiol. 2010;63(12):1281–1282. doi:10.1016/j.jclinepi.2010.06.005
51.		Oh MK, Jo H, Lee YK. Improving the reliability of clinical practice guideline appraisals: effects of the Korean AGREE II scoring guide. J Korean Med Sci. 2014;29(6):771–775. doi:10.3346/jkms.2014.29.6.771
52.		Brouwers MC, Kerkvliet K, Spithoff K, AGREE Next Steps Consortium. The AGREE Reporting Checklist: a tool to improve reporting of clinical practice guidelines. BMJ. 2016;352:i1152. doi:10.1136/bmj.i1717
53.		Keating D, McWilliams S, Schneider I, et al. Pharmacological guidelines for schizophrenia: a systematic review and comparison of recommendations for the first episode. BMJ Open. 2017;7:e013881. doi:10.1136/bmjopen-2016-013881
54.		Münzberg M, Mutschler M, Paffrath T, et al. Level of evidence analysis for the latest German National Guideline on treatment of patients with severe and multiple injuries and ATLS. World J Surg. 2015;39(8):2061–2067. doi:10.1007/s00268-015-3063-z
55.		Burda BU, Norris SL, Holmer HK, Ogden LA, Smith ME. Quality varies across clinical practice guidelines for mammography screening in women aged 40–49 years as assessed by AGREE and AMSTAR instruments. J Clin Epidemiol. 2010;64(9):968–976. doi:10.1016/j.jclinepi.2010.12.005
56.		Sandström B, Willman A, Svensson B, Borglin G. Perceptions of national guidelines and their (non) implementation in mental healthcare: a deductive and inductive content analysis. Implement Sci. 2015;10:43. doi:10.1186/s13012-015-0234-0
57.		Ince P, Haddock G, Tai S. A systematic review of the implementation of recommended psychological interventions for schizophrenia: rates, barriers, and improvement strategies. Psychol Psychother. 2016;89(3):324–350. doi:10.1111/papt.12084
58.		Jun J, Kovner CT, Stimpfel AW. Barriers and facilitators of nurses’ use of clinical practice guidelines: an integrative review. Int J Nurs Stud. 2016;60:54–68. doi:10.1016/j.ijnurstu.2016.03.006
59.		Chou AF, Vaughn TE, McCoy KD, Doebbeling BN. Implementation of evidence-based practices: applying a goal commitment framework. Health Care Manage Rev. 2011;36(1):4–17. doi:10.1097/HMR.0b013e3181dc8233
60.		McGowan J, Muratov S, Tsepke A, Issina A, Slawecki E, Lang ES. Clinical practice guidelines were adapted and implemented meeting country-specific requirements – the example of Kazakhstan. J Clin Epidemiol. 2016;69:8–15. doi:10.1016/j.jclinepi.2015.05.001
61.		Saeed SA, Bloch RM, Silver S. Role of leadership in narrowing the gap between science and practice: improving treatment outcomes at the systems level. Psychiatr Q. 2015;86(3):311–323. doi:10.1007/s11126-015-9372-4
62.		Manikam L, Hoy A, Fosker H, et al. What drives junior doctors to use clinical practice guideline? A national cross-sectional survey of foundation doctors in England and Wales. BMC Med Educ. 2015;15:227. doi:10.1186/s12909-015-0510-3
63.		Pronovost P. Enhancing physicians’ use of clinical guidelines. JAMA. 2013;310(23):2501–2502. doi:10.1001/jama.2013.281334
64.		Quanbeck A, Brown RT, Zgierska AE, Johnson RA, Robinson JM, Jacobson N. Systems consultation: protocol for a novel implementation strategy designed to promote evidence-based practice in primary care. Health Res Policy Syst. 2016;14:8. doi:10.1186/s12961-016-0157-5
65.		Joosen MC, van Beurden KM, Terluin B, van Weeghel J, Brouwers EP, van der Klink JJ. Improving occupational physicians’ adherence to a practice guideline: feasibility and impact of a tailored implementation strategy. BMC Med Educ. 2015;15:82. doi:10.1186/s12909-015-0364-8
66.		Westbrook JI, Ampt A, Kearney L, Rob MI. All in a day’s work: an observational study to quantify how and with whom doctors on hospital wards spend their time. Med J Aust. 2008;188(9):506–509.
67.		Westbrook JI, Duffield C, Li L, Creswick NJ. How much time do nurses have for patients? A longitudinal study quantifying hospital nurses’ patterns of task time distribution and interactions with health professionals. BMC Health Serv Res. 2011;11:319. doi:10.1186/1472-6963-11-319
68.		Gold R, Hollombe C, Bunce A, et al. Study protocol for “Study of Practices Enabling Implementation and Adaptation in the Safety Net (SPREAD-NET)”: a pragmatic trial comparing implementation strategies. Implement Sci. 2015;16:144. doi:10.1186/s13012-015-0333-y
69.		Grimmer K, Machingaidze S, Dizon J, Kredo T, Louw Q, Young T. South Africa clinical practice guidelines quality measured with complex and rapid appraisal instrument. BMC Res Notes. 2016;9:244. doi:10.1186/s13104-016-1938-1
70.		Kortteisto T, Laitila M, Pitkänen A. Attitudes of mental health professionals towards service user involvement. Scand J Caring Sci. 2017;32(2):681–689. doi:10.1111/scs.12495.
71.		Bachmann LM, Mühleisen A, Bock A, ter Riet G, Held U, Kessels AGH. Vignette studies of medical choice and judgement to study caregivers’ medical decision behavior: systematic review. BMC Med Res Methodol. 2008;8:50. doi:10.1186/1471-2288-8-50