Evaluating the impact of national education in pediatric palliative care: the Quality of Care Collaborative Australia

Introduction

Based on data from the UK,¹ it is estimated that 32 per 10,000 children in Australia are living with a life-limiting condition. Most specialist pediatric palliative care services (PPCS) are based in capital cities in Australia and provide outreach via telehealth.^2,3 In Queensland (QLD) alone, 150 patients were referred to the tertiary PPCS based in Brisbane over a 2-year period from July 2009, 58% of which were from regional or rural areas.⁴

The needs of families and communities supporting children with life-limiting conditions are complex and often ongoing, and health professionals and other agencies require specialized knowledge and skills to provide effective care. As there are no dedicated PPCS outside of tertiary hospitals in Australia, children and their families rely on a network of existing local health professionals for care. This can include pediatric services (eg, child health nurse), palliative care services (adult and community), and other public and private community services. The tertiary PPCS are available for consultation and advice. The 1:60 ratio of child-to-adult deaths in Australia indicates that health care professionals who routinely care for adults with palliative and end of life care needs require extra support to provide care for children due to the infrequency in presentation to their services.⁴ This is particularly the case for health professionals in regional and remote areas. Consideration also needs to be given to the significant impact that the death of a child has on the family, community, and health professionals involved.

To address the educational needs of these health professionals, a national collaboration titled the Quality of Care Collaborative Australia (QuoCCA) was funded under the Australian Government Department of Health, National Palliative Care Projects, to provide national education in pediatric palliative care (PPC) from May 2015 to the end of June 2017 initially.⁵

This education was coordinated through Nurse Educators, Medical Fellows, a national Allied Health Educator and national project staff based in the tertiary PPCS in the following hospitals: Queensland Children’s Hospital (Brisbane, QLD), Sydney Children’s Hospital (Randwick, New South Wales [NSW]), John Hunter Children’s Hospital (Newcastle, NSW), Royal Children’s Hospital (Melbourne, Victoria [VIC]), Women’s and Children’s Health Network (Adelaide, South Australia [SA]), and Perth Children’s Hospital (Perth, Western Australia [WA]). These teams also provided education to the Northern Territory (NT), Tasmania (TAS), and Australian Capital Territory (ACT), thus covering all of Australia. The activity was governed via a National Steering Committee and managed through a lead entity at Children’s Health Queensland (CHQ). A learning needs analysis was conducted to guide the content of the education.⁶

The goal of QuoCCA was to achieve service quality improvement in national PPC in acute and community settings, through education, research, and evaluation within a quality framework, building on the foundation of the work of the national collaborative. The objectives were to have:

• Improvement in skill, knowledge, and confidence of national health professionals at all levels involved in the delivery of PPC enabling them to provide best practice care for children and families.
• Improvement in timely and appropriate access to, and quality of, PPC in national acute and community settings.
• Improved processes for care planning in local communities.
• Increased community and health professional awareness of death and dying as a normal part of life-limiting conditions in childhood and the need for family centered and developmentally and culturally appropriate services.

The reach included health professionals involved in acute and community care and those working in regional, rural, and remote locations throughout Australia through a program of education and a suite of resources.

Existing clinical education standards recommend evaluating four levels: 1) process evaluation regarding participant satisfaction; 2) impact evaluation regarding acquisition of knowledge, skills, or confidence; 3) outcome evaluation regarding changes in practice; and 4) service quality improvement.^7,8 Based on this, a project logic was developed that detailed the outputs, impacts, and longer term outcomes required of the project, with the associated evaluation tools.

This article examines the impact of QuoCCA education through the use of pre- and post-education surveys completed by health professionals and other participants during the first funding round of the project, with education being provided between May 2015 and August 2017. The evaluation plan for QuoCCA education also included interviews with health professionals, educators, and families and an outcome survey examining changes in practice and long-term improvement in knowledge and confidence, which will be reported on in future papers.

Methods

The QuoCCA funded educators (medical, nursing, and allied health) were experienced PPC providers who had various degrees of experience in the delivery of PPC education. In addition, other medical, nursing, and allied health staff from tertiary services worked alongside QuoCCA educators, bringing their expertise to the sessions. This reflected the multi-disciplinary approach to PPC. QuoCCA provided education above a measured baseline of education, which was already being provided by the tertiary PPCS in each state.

The educators worked together to construct standard curricula, through regular meetings to discuss the education content, quality, and effectiveness; sharing resources and presentations through a national secure file transfer system; and the educators traveling to other states to participate in education sessions.

The content of education was flexible for each audience to maximize the relevance against current needs. Educators had a collection of baseline presentations from which they chose the most appropriate for the audience.

An introduction to PPC agenda would normally include what is PPC, what/who is the PPCS, myths and dilemmas, pain and symptom management (including nonpharmacological strategies), communication, contemporary grief and bereavement concepts, spirituality, case studies, end of life care, and self and team care.

More advanced topics were delivered as relevant to the audience and/or the patient(s) in the area and included advance care planning, cultural/spiritual concerns, transformational care and bereavement, disease-specific information, age-specific information (eg, neonates, adolescents, and young adults), end of life care, allied health specific information, ethics, and practical considerations (eg, disability, equipment, and respite).

The majority of education sessions conducted by QuoCCA were evaluated with pre- and post-education surveys with questions measuring the confidence or knowledge related to nine dependent variables (Supplementary materials). These variables (and shortened names) were as follows: the management of new referrals for PPC patients (referrals), symptom management along five domains (“pain”, “nausea”, “dyspnea”, “seizures”, and “fear or anxiety” [fear]), confidence in helping families prepare for the death of their child (preparation), knowledge of resources such as agencies that can assist in providing care (resources), and confidence giving medications (medication).

The four confidence questions had a Likert scale of 1–5 from 1= not at all confident to 2= slightly confident, 3= moderately confident, 4= very confident, and 5= extremely confident, or N/A = not applicable. The knowledge questions had a Likert scale of 1–5 from 1= no knowledge to 2= slight knowledge, 3= moderate knowledge, 4= high level of knowledge, and 5= extreme level of knowledge, or N/A = not applicable.

The survey also collected information on eight independent variables, which were used as collected or re-coded as follows:

1. Education type – this was categorized as scheduled, pop-up, or incidental. Scheduled education was pre-planned general PPC education in any location. Pop-up education was related to the care of a specific patient, supporting local services away from a tertiary center to care for the child and family. In this education type, a small inter-professional team of health professionals traveled to a center to deliver education, responding in a timely way to patient/family needs.⁹ Incidental education was provided as a part of other meetings or activities in a tertiary center (eg, handover, multi-disciplinary, and morbidity and mortality meetings).
2. Experience – previous involvement in the care of a child with a life-limiting condition (yes/no).
3. Previous education – previous attendance at education days conducted by the tertiary PPCS in their state (yes/no).
4. Remoteness – this was based on the location of the education as determined by the Australian Statistical Geography Standard remoteness structure¹⁰ and accessed through Doctor Connect.¹¹
5. State – this was initially coded as the state in which the education was conducted. However, as there were a smaller number of surveys from the smaller states without a QuoCCA educator, this was re-coded as the state who conducted the education. ACT was recoded to QLD, NT was recoded to SA, and TAS was recoded to VIC, representing the states that mostly provided QuoCCA educators to conduct this education. Each of the five states with QuoCCA educators conducted education in their own state, although on occasion they were accompanied by QuoCCA educators from other states.
6. Length – the length of education session was re-coded as short = sessions of 2 hours or less, half-day = 3–4 hours and full-day = more than 4 hours.
7. Financial year – this was recoded to combine 2014/2015 and 2015/2016 due to the occurrence of limited sessions in the former; 2016/2017 stood alone.
8. Occupation – nurse, doctor, social worker, occupational therapist, physiotherapist, pharmacist, psychologist, music therapist, and others (specified).

Following education sessions, QuoCCA educators completed a report on the session through an online survey, including date, location, educators, attendees, and open text comments on the following:

• What is your feedback on how this session went?
• What lessons did you learn from this education session?

Qualitative data related to the topics and the effectiveness of education were analyzed using an inductive thematic approach.¹² Educators were also asked for feedback at the end of the project.

Ethics

The protocol for the evaluation of this newly developed national collaborative in education was approved by the Children’s Health Queensland Human Research and Ethics Committee (HREC/16/QRCH/60). Participants were provided with information about the project and advised that participation was not mandatory and all questionnaires completed were anonymous.

Statistical analyses

The recommended sample size for a population size of 5,773, with a confidence level of 95% and a margin of error of 5%, is 361. We had paired surveys completed by 969 participants.

Descriptive statistics of the dependent variables was reported as median and inter-quartile range (IQR) for non-normally distributed continuous data. Normality was assessed using a Shapiro–Wilk test, which indicated that none of the variables were normally distributed (P<0.001). Frequencies (raw counts and percentages) were presented for categorical data. Proportions of categorical data were compared using a Chi-squared test, eg, occupation, previous education, and experience.

Pre- and post-education surveys were paired by participant. A nonparametric Wilcoxon signed ranks’ test was used to assess the difference between pre- and post-education scores for each of the dependent variables (referral, pain, nausea, dyspnea, seizure, fear, resources, preparation, and medication).

Change scores in paired education surveys were calculated for each dependent variable by subtracting the post-score from the pre-score when information was collected at both time points for individual participants. These change scores were analyzed against the independent variables to determine if they influenced improvement following education. A Mann–Whitney U test was used for independent variables with two categories (experience, previous education, and financial year). A Kruskal–Wallis test was used (along with Dunn’s post hoc test with multiple testing adjustment when the overall result was significant) for independent variables with more than two categories (education type, state, length, occupation, and remoteness).

In addition, the proportion of participants who showed an improvement in scores between pre- and post-education surveys (a change score of 1 or more), were compared with categories of experience and education using a chi-squared test.

Finally, a binomial logistic regression was performed to investigate factors that could potentially predict improvement following education. This involved re-coding the change score for each dependent variable into a binomial variable indicating improvement (a difference between pre- and post-scores of 1 or more) or no improvement (a difference of less than 1). The analysis was performed on the categorical independent variables using the following as the reference points – having experience caring for children with life-limiting conditions, having had previous education, financial year of 2014/2015 or 2015/2016 (vs 2016/2017), occupation of nursing (vs allied health and other), and major city (vs regional/remote). The odds ratio (OR) and the P-value were included where they were significant.

All statistical analyses were performed using SPPS Version 24, and P-values<0.05 were considered statistically significant.

Results

What were the outputs of the education?

From May 2015 to June 2017, there were 337 education sessions conducted by the QuoCCA Project, which included 183 scheduled, 107 pop-up, and 47 incidental sessions, totaling 767 hours and delivered to 5,773 attendees. Education was delivered in all states and territories of Australia, and the remoteness of the locations is shown in Table 1.

Table 1 Characteristics of QuoCCA education sessions by remoteness Abbreviation: QuoCCA, Quality of Care Collaborative Australia.

The mean number of participants in scheduled education sessions was 20.5 (maximum 274), 10.3 for pop-up sessions (maximum 38), and 12.7 for incidental sessions (maximum 50). There was generally a greater percentage of pop-up education sessions in the more regional and remote areas and a greater number of recorded participants per session in remote areas (Table 1).

A range of health professionals received education including medical (n=808), nursing (n=3,280), allied health (n=562), and others (n=617) such as administration staff, aboriginal health workers, community development workers, chaplains and pastoral carers, teachers, funeral directors, and paramedics (note that 506 participants did not report their occupation). Health professional participants included students, managers, educators, and research staff. Allied health professions represented were occupational therapy, speech pathology, dietetics, music therapy, pharmacy, physiotherapy, psychology, radiation therapy, and social work. Dentists also attended the education.

Examining the attendance of the different occupations at QuoCCA education, the pop-up sessions had a greater proportion of medical staff compared to other education types (14% medical, 74% nursing, and 11% allied health) (P=0.02). In scheduled education sessions, there was no significant difference in the proportion of different occupations who attended full-day, half-day, or short sessions (P=0.29).

By occupation, a greater percentage of nurses had received previous education and had experience in caring for a child with a life-limiting condition. A greater percentage of medical and allied health staff had no previous education (P=0.002) or experience (P=0.04).

Did participants report a change in confidence/knowledge following education?

Twenty-six percent (n=1,474) of participants submitted either a pre-education survey or post-education survey, and 17% (n=969) submitted both pre- and post-surveys. Only these paired survey responses were analyzed. Not all education sessions had surveys collected. Of the 1,100 participants at pop-up sessions, 9% of the participants completed both pre- and post-surveys. This is compared with 22% of the 3,747 scheduled education participants and 5% of the 547 incidental participants completing both surveys.

Table 2 shows the median and IQR of the pre- and posteducation scores, as well as the change score, for all nine dependent variables. For all measures, there was a significant increase in knowledge or confidence following education (P<0.001).

Table 2 Result of univariate analyses on the pre- and post-education scores

Table 3 shows the number of paired surveys that showed a negative change score between pre- and post-education, a positive change score (an improvement in confidence or knowledge), and no difference. The questions related to the management of referrals, preparation for the death of a child, and resources showed the largest percentage of participants with an improvement in knowledge or confidence. Only 2%–6% of participants showed a decline in change score following education.

Table 3 The number of participants showing a decline, no change or improved scores from pre-education to post-education surveys for each variable measured

Which independent variables had an impact on the education change score?

Change scores (the difference between post- and pre-education survey scores) for all measures were analyzed against the categories of each independent variable (Table 4).

Table 4 Result of univariate analyses (P-values) on the impact of independent variables on change scores (difference between pre- and post-education scores) for all measures Notes: P-values from the statistical tests that examined how the independent variables impacted on the difference between pre-education and post-education measures. *P<0.05. **P<0.01. ***P<0.001.

Participants with no experience in caring for a child with a life-limiting condition had significantly higher change scores for all measures except fear and medication (Table 4, Mann–Whitney U test, P<0.05). In addition, when comparing each of the measures for those participants with experience and those without, a significantly greater percentage of participants without experience showed an improvement in post-education scores except for dyspnea, fear, resources, and medication (Chi-squared, P<0.05; Table 5).

Table 5 Percentage of participants who reported improved change scores following education (increase of one or more in scores) and median values (IQR) for pre- and post-education scores by experience in caring for children with life-limiting conditions

Although those with no experience had relatively higher change scores, so showed greater improvement, their actual median scores were lower than those with experience (Table 5). The participants with experience scored themselves significantly higher before the education session (Mann–Whitney for prescores, P<0.001; Table 5) and then reported higher post-education knowledge or confidence across the measures of the education (except for resources) (Mann–Whitney, P<0.01; Table 5).

Participants with no previous education in PPC had significantly higher change scores for all measures (Table 4, Mann–Whitney U test, P<0.05). In addition, a significantly higher percentage of participants who had no previous education showed an improvement in post-education scores for all measures except medication (Chi-squared, P<0.05; Table 6).

Table 6 Percentage of participants with and without previous education who reported improved change scores following education (% 1+) and median values (IQR) for pre- and post-education scores

Those with no previous education showed a greater improvement in their change scores, although their median pre- and post-scores were lower than those with previous education (Table 6). Participants with previous education scored themselves significantly higher before the education session (Mann–Whitney for prescores, P<0.05; Table 6) and then generally reported higher knowledge or confidence following education across the measures of the education (except for pain, nausea, and dyspnea) (Mann–Whitney, P<0.05; Table 6).

There was no significant difference (P>0.05) in change scores in different financial years except for knowledge regarding resources, where 2016/2017 education sessions showed a greater improvement between pre- and post-scores (P=0.001; Table 4).

The type of education provided no significant impact on the change score, except for confidence in managing nausea (P=0.042), where pop-up education showed a higher mean rank for the difference between pre- to post-education scores (Table 4). There was also no significant difference in the percentage of participants that showed improved scores by education type (P>0.05).

There was no significant impact of occupation on the change scores for any measure (P>0.05; Table 4).

The length of the education session had an impact on the change scores for management of referrals (P=0.023; Table 4), pain (P<0.001), nausea (P=0.002), and fear (P=0.01). There were higher rankings of change scores in full-day education for pain, nausea, and fear and half-day education sessions for management of referrals. Short education sessions showed significantly lower change score rankings for referrals, pain, nausea, and fear.

Table 7 shows the percentage of participants that improved by the length of education session, which was significant for all variables except dyspnea and seizures (P<0.05). In all the significant variables, short education sessions showed a smaller proportion of participants whose scores had improved, followed by half-day education and then full-day education (Table 7). In total, for all variables, 55% showed improved scores following short education, 63% for half-day and 66% for full-day.

Table 7 Percentage of participants who showed an improvement in knowledge or confidence by length of the education session

The only measure significantly impacted by remoteness was the knowledge of resources (P=0.006). Very remote education sessions showed higher change scores related to resources, followed by major city sessions.

The state that conducted the education had a significant impact on the pre- and post-education scores for all measures except for medication (Kruskal–Wallis test, P<0.05). SA had the highest ranking of pre- to post-education scores for referrals, pain, seizures, fear, resources, and preparation. QLD had the higher ranking for dyspnea.

Can we predict an improvement following education?

The results of the binomial logistic regression are shown in Table 8. Experience, previous education, length and type of education session, and state conducting the education were significant for most variables, and financial year of the education was significant for knowledge about resources (Table 8).

Table 8 Binomial logistic regression of improvement (yes, no) in confidence/knowledge between pre- and post-education surveys, showing the model’s P-value, odds ratio and 95% confidence interval. Notes: Where the model was significant (P<0.05), the model P-value and OR (95% CI) are listed. No comparisons for medication, education type, or occupation (nursing vs other) were significant.

Participants with no experience in caring for children with a life-limiting condition were 1.9–2.7 times more likely to improve in post-education scores as compared with those with such experience across the dimensions of referral, pain, nausea, seizures, and preparation for death. Note that those with experience started with a higher pre-education score and also had a higher score following education (P<0.05; Table 5).

Participants with no previous education in PPC were 1.5–2.5 times more likely to improve in post-education scores as compared with those with previous education across the dimensions of referrals, pain, nausea, dyspnea, seizures, fear, resources, and preparation for death (that is all dimensions measured except medication). Those with previous education started with a higher pre-education score and had a higher score following education (P<0.05; Table 6).

Participants in the financial year 2016/2017 were 1.7 times more likely to improve in the knowledge of resources, compared to the earlier 2014/2016 participants.

Regional and remote participants were 1.4 times more likely to improve than city participants in confidence with the management of nausea and seizures.

Regarding the length of the education session, full-day education participants were 1.5–2.2 times more likely to improve compared to short day education participants along the dimensions of referral, pain, nausea, fear, resources, and preparation (conversely, participants in short education sessions were 0.5–0.7 times likely to improve compared to full-day education). Half-day education participants were 1.6–1.7 times more likely to improve compared to short education sessions along the dimensions of pain and nausea (conversely, short education sessions were 0.6 times more likely to improve compared to half-day education).

What education topics did participants request?

Table 9 shows the education topics that were requested by the participants in the surveys and the numbers of participants that requested them. There were six major themes in the topics requested related to clinical care (with sub themes of family, treatment, symptoms, processes, and others), bereavement, health care team/services, communication, staff well-being, and education.

Table 9 Topics that participants hoped to hear (pre-education) and would like to be covered in future (post-education survey) Abbreviation: PPCS, Pediatric Palliative Care Service.

The participants commented about the modes of education that they found most effective. These included case studies, parent and child perspectives, simulations, mythbusters and FAQ, videos, online learning resources, and copies of the PowerPoint slides.

Two participants suggested education targeted to various professions, particularly allied health professions. Presentations from parents and patient stories were very effective. Thirteen participants wanted more frequent, such as annual, education in the subject area.

What was the educators’ feedback on the effectiveness of education?

There were 303 comments provided by the QuoCCA educators in their reports on education sessions and what lessons they learnt, 50 of which were related to the effectiveness of education. Educators found they should tailor the presentation to the audience both in content and style. Learning needs of the group were ascertained up front, by gathering audience concerns and questions or by having a discussion with a local champion prior to the session.

Good outcomes were evident when participants shared their experiences of palliative care, including “pair and share” items, when audience examples were sought, and the educator deferred to the knowledge in the room.

One educator had a pattern for half-day or longer workshops, where a case study would flow into a role play, involving group work. An overview of PPC and the tertiary service would include story telling. Another educator stated that audience participation and knowledge sharing in the session had a vital role to play in the effectiveness of education. This also took the pressure off the educator needing to be everybody’s expert. For short sessions of an hour or less, educators would consider having an informal discussion rather than a slide presentation, which encouraged audience participation and directly addressed their needs.

Discussion

QuoCCA education was delivered to a range of health professionals and other service providers in each state and territory around Australia. The variety of participants reflected the response required in the community to care for a child receiving palliative care and their family, including community services, hospital services, schools, pastoral carers, primary health carers and funeral directors.

The QuoCCA national collaboration provided opportunities for educators based in tertiary PPCS from different states to develop a common curriculum, share information, deliver education sessions together as a team, and discuss their experiences of effective education. This included the use of innovative and interactive education methods such as case-based discussion, role play, simulation, and the use of video and artwork. In this way, the QuoCCA collaboration contributed to the quality of education delivered to health professionals to improve learning outcomes and consequently improve the quality of PPC services throughout Australia. This was also the experience in centers working together to improve care quality and reduce variation in the care of children with hypoplastic left heart syndrome.¹³

The evaluation scores showed a significant improvement in knowledge or confidence in all the measures following education. A large percentage of participants had improved education scores for confidence in managing new referrals, preparing the family for the death of their child and knowledge of resources such as agencies that can assist. These broad dependent variables were less task orientated than those related to symptom management and giving medication, which were case driven and more challenging to deliver in a workshop setting.

We were able to predict which participants were more likely to show an improvement in their scores following education. They were those who had no experience for caring for a child with a life-limiting condition, those who had no previous education in PPC, and those who attended full-day or half-day education rather than short education sessions. In one or two measures, those who received education in the latter half of the project, and those from regional or remote areas showed a greater improvement.

Our results demonstrate that novice and experienced professionals build confidence and knowledge from participation in ongoing professional education. In fact, the higher dosage of education in terms of the length of the session and previous attendance at education resulted in increased knowledge and confidence. This result was also shown in other studies, where prior preparation for education and postgraduate qualifications in the area improved the impact of education.^14–16 In Taiwan, the frequency of contact with palliative care services and further education were predictors of knowledge and attitude to a palliative care consultative service.¹⁷ Importantly, they found that access to a palliative care team for consultation regarding patients had a positive impact on the care provided. In another study, there was a threshold of palliative care education of 40 hours, beyond which physicians were less likely to recommend major operative interventions, implying that there was a dosage of education that was necessary to impact practice.¹⁸ Future evaluation in pediatric palliative care education could analyze the amount of education received by individuals, and the impact that has on their practice.

Ferguson et al¹⁹ stated that most education programs in PPC included symptom management and pain control. Our QuoCCA participants requested education along the themes of clinical care, staff well-being available services, communication, interprofessional roles, and bereavement. Within these themes, the specific education topics most requested were management of medications, bereavement of families and staff, family support, symptom management, communication, end of life management, emotional management, utilization of local services, and cultural differences. These topics aligned with the 12 most highly rated learning needs found by the QuoCCA project according to weighted average, which were end of life management, communication skills, symptom management, bereavement care, emotional supports, grief and loss, and self-care.⁶

Traditionally, education programs in PPC have taught knowledge and skills through lecture and presentation style education.¹⁹ A review of the impact of education on clinician practice and patient outcomes found interactive techniques to be most effective, while didactic presentations and printed information had little or no benefit.²⁰ Interactive education that used blended innovative methods such as multimedia, case studies, role play, small group discussion, and simulations is known to be effective.^21–28 Improved learning outcomes occurred with reflective questions, prior planning of education, quality improvement activities incorporated into the education,²⁹ as well as the use of a human patient simulator³⁰ and simulations.³¹ The American Nurses Credentialing Center’s Commission on Accreditation³² also reported that the transfer of new knowledge and skills was facilitated through simulation, interactive computer programs, and web-based applications. The experience of the QuoCCA educators was that the most effective education was more interactive and involved greater participation and reflection. QuoCCA education focused on promoting the palliative care approach. Such concepts can only be transferred to health professionals through education that facilitates reflection and learning from the experience of families, children, and other health professionals.

Participants in the longer QuoCCA education sessions (full-day or half-day) showed higher likelihood of improvement in knowledge or confidence. A review of continuing medical education also found a positive correlation between education effectiveness and the length of interventions.²¹

The QuoCCA education sessions benefited from a small number of participants per session^21,23 and the provision of additional structured resources for self-directed learning following the education session via a QuoCCA website (https://www.caresearch.com.au/quocca/).²⁵

Limitations

The study methodology incorporated surveys in which participants self-reported their level of knowledge or confidence. However, participant’s self-awareness of their individual knowledge or skills, and various influences on their confidence level, may impact on the effectiveness of these surveys.^33–36 For example, an experimental design with a control group found that extraneous variables may have interfered with measures of nursing quality.³⁷

The surveys examined generic areas of the application of PPC education that would improve the management of new referrals, symptom management, medications, preparing the family, and using local agencies. Effectively the analysis examined the increase in knowledge and confidence in as much as it fulfilled the gaps as perceived by the audience, enabling them to provide higher quality PPC in these areas. With any education, the perception of increase in knowledge or confidence as assessed by the participant will be a reflection of how the education has met their expectations and brought them forward in their knowledge and confidence in delivering the target service.

Survey collection was not consistent for all education sessions. Not all the education sessions had pre- and/or post-evaluation performed, and for practical reasons, the shorter and pop-up education sessions were less likely to have surveys collected. This may have skewed the results for the short education sessions. There may also be some reporter bias in the participants who were more likely to complete a survey.

Between 2% and 6% had decreased scores in the various measures following education and reasons for this could vary. It is possible that after the education, the participant realized they did not know as much as they initially thought. Another possibility is that they simply did not take into account the score they put in the pre-survey when they answered the post-survey. Alternatively, they could have been dissatisfied with the education and decided to indicate that in their scores.

It is recommended that the goals of evaluation should be clearly linked to the required outcomes of the teaching.³⁸ However, the goals of each QuoCCA education event were determined by the learning objectives of participants at each session. Furthermore, pop-up education was specific to a patient’s and family’s needs, their supporting services, and the incidental education specific to its context. Thus, the curriculum provided was not standard across educators or states but tailored to the audience and the needs of the service.

The same pre- and post-education surveys were used across the national project and were not tailored to the education session. However, the actual content of the education sessions varied. For example, not all symptom management and medication measures were covered in all sessions and this was reflected in lower improvement scores for some of these measures. The surveys for the second round of QuoCCA funding (2017–2020) will be combined to a single post-education survey with more generic measures relevant to PPC and include an option to tick when the session did not cover that topic.

Generally, the educational impact of QuoCCA sessions may have been influenced by disparate education models and methods and characteristics of the audience receiving that education. The national collaborative will continue to provide benefit through ongoing harmonization of the curriculum and teaching methodology throughout Australia.

Conclusion

QuoCCA provided education to over 5,770 participants during the 2.2 years of education delivery in this initial project. The evaluation of education sessions found significant improvements in scores following education in all the measures collected. Longer education sessions resulted in greater improvement in confidence or knowledge. Participants who did not have experience in caring for children with a life-limiting condition and those with no previous education in PPC showed higher improvement in scores from pre- to post-surveys. Although their overall improvement was less, experienced staff had higher scores in both their pre- and post-education surveys. Therefore, these results encourage the continuation of the provision of education to both novice and experienced professionals and others who care for children with a life-limiting condition, moving both to higher confidence and knowledge.

The learnings from this evaluation will be transferred into the second round of funding for this national education project, QuoCCA 2. The next stage will continue to deliver education and focus on quality improvement methodologies, simulation and interactive training, accessible harmonized training modules, and videos on the national website (www.caresearch.com.au/quocca/, hosted by CareSearch). The project will develop and share a body of knowledge to raise the awareness of the palliative care approach and the benefits of timely referral to local services that provide good quality PPC.

Acknowledgments

This project involved a collaboration of six tertiary pediatric palliative care entities throughout Australia managed through the National Project Lead Entity, Children’s Health Queensland (CHQ) Hospital and Health Service, headed by the National Project Sponsor, Chief Executive CHQ, Fionnagh Dougan. The collaboration included staff from tertiary Pediatric Palliative Care Services from Queensland Children’s Hospital, Brisbane, Sydney Children’s Hospital, Randwick, John Hunter Children’s Hospital, Newcastle, Royal Children’s Hospital, Melbourne, Women’s and Children’s Health Network, Adelaide, and Perth Children’s Hospital, Perth. Appreciation goes to the collaboration members and their supporting agencies, including the Project Leads from each state – Sara Fleming, South Australia, Jenny Hynson, Victoria, Marianne Phillips, Western Australia, Suzanne Momber, Western Australia, Sharon Ryan, NSW, and Susan Trethewie, NSW. Anne Bernard, senior biostatistician from QFAB Bioinformatics, provided excellent guidance in the development of the statistical analyses. Many thanks also to the participants in the QuoCCA education from both the private and public sectors and their helpful assistance in completing the evaluation tools. The QuoCCA Project was undertaken through Australian Government Department of Health funding provided from the National Palliative Care Projects operating under the Chronic Disease Prevention and Service Improvement Fund, grant funding round H1314G012.

Disclosure

The authors report no conflicts of interest in this work.

References

1.		Fraser LK, Miller M, Hain R, et al. Rising national prevalence of life-limiting conditions in children in England. Pediatrics. 2012;129(4): e923–e929.
2.		Hynson J, Drake R. Paediatric palliative care in Australia and New Zealand. In: Knapp C, Madden V, Fowler-Kerry S, editors. Pediatric Palliative Care: Global Perspectives. Heidelberg: Springer; 2012:379–402.
3.		Bensink ME, Armfield NR, Pinkerton R, et al. Using videotelephony to support paediatric oncology-related palliative care in the home: from abandoned RCT to acceptability study. Palliat Med. 2009;23(3):228–237.
4.		Herbert A, Bradford N, Donovan L, Pedersen LA, Irving H. Development of a state-wide pediatric palliative care service in Australia: referral and outcomes over two years. J Palliat Med. 2014;17(3):288–295.
5.		Herbert A, Irving H, Pedersen LA, et al. Quality of care collaborative for paediatric palliative care in Australia (QuoCCA). Pediatric Blood and Cancer, Conference, 30th Annual Meeting of the American Society of Pediatric Hematology/Oncology, ASPHO 2017, Canada; 2017;64:S65. doi: 10.1002/pbc.26591.
6.		Baggio S, Herbert A, Delaney A, et al. A national quality of care collaboration to improve paediatric palliative care outcomes. Proceedings of the 14th National Rural Health Conference, editor Leanne Coleman, Cairns, Queensland, 26–29 March 2017. Canberra: National Rural Health Alliance; 2017.
7.		Ferguson A. Evaluating the purpose and benefits of continuing education in nursing and the implications for the provision of continuing education for cancer nurses. J Adv Nurs. 1994;19(4):640–646.
8.		Kirkpatrick D. “The Four Levels of Evaluation”. Evaluating Corporate Training: Models and Issues. Brown S, Seidner C, editors. Norwell, MA: Kluwer Academic; 1998:95–112.
9.		Mherekumombe MF, Frost J, Hanson S, Shepherd E, Collins J. Pop Up: A new model of paediatric palliative care. J Paediatr Child Health. 2016;52(11):979–982.
10.		Pink B [homepage on the Internet]. Australian Statistical Geography Standard (ASGS): Volume 5 – Remoteness Structure. Australia. July 2011. Australian Bureau of Statistics, Catalogue No. 1270.0.55.005. Canberra: Commonwealth of Australia; 2013. Available from: http://www.abs.gov.au/AUSSTATS/[email protected]/Lookup/1270.0.55.005Main+Features1July%202011?OpenDocument. Accessed July 17, 2018.
11.		Doctor Connect [homepage on the Internet]. Canberra: Australian Government, Department of Health. Doctor Connect. 2018. Available from: http://www.doctorconnect.gov.au/internet/otd/publishing.nsf/Content/locator. Accessed July 17, 2018.
12.		Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101.
13.		Clauss SB, Anderson JB, Lannon C, et al. Quality improvement through collaboration: the national pediatric quality improvement collaborative initiative. Curr Opin Pediatr. 2015;27(5):555–562.
14.		Bird A, Wallis M. Nursing knowledge and assessment skills in the management of patients receiving analgesia via epidural infusion. J Adv Nurs. 2002;40(5):522–531.
15.		Tippett J. Nurses’ acquisition and retention of knowledge after trauma training. Accid Emerg Nurs. 2004;12(1):39–46.
16.		Considine J, Botti M, Thomas S. The effects of specific educational preparation on emergency nurses’ clinical decisions regarding supplemental oxygen administration. Nurs Health Sci. 2006;8(2):73–80.
17.		Pan HH, Shih HL, Wu LF, Hung YC, Chu CM, Wang KY. Path modeling of knowledge, attitude and practice toward palliative care consultation service among Taiwanese nursing staff: a cross-sectional study. BMC Palliat Care. 2017;16(1):42.
18.		Bateni SB, Canter RJ, Meyers FJ, Galante JM, Bold RJ. Palliative care training and decision-making for patients with advanced cancer: A comparison of surgeons and medical physicians. Surgery. 2018;164(1):77–85.
19.		Ferguson LM, Fowler-Kerry S, Hain R. Education and Training. In: Goldman A, Hain R, Liben S, editors. Oxford Textbook of Palliative Care for Children. Oxford: Oxford University Press; 2006:594–614.
20.		Bloom BS. Effects of continuing medical education on improving physician clinical care and patient health: a review of systematic reviews. Int J Technol Assess Health Care. 2005;21(3):380–385.
21.		Mansouri M, Lockyer J. A meta-analysis of continuing medical education effectiveness. J Contin Educ Health Prof. 2007;27(1):6–15.
22.		Marinopoulos SS, Dorman T, Ratanawongsa N, et al. Effectiveness of Continuing Medical Education. Evidence Reports/Technology Assessments, No. 149, Report No.: 07-E006. Rockville (MD): Agency for Healthcare Research and Quality (US); 2007.
23.		Ghosh AK. Organizing an effective continuous medical education session. J Assoc Physicians India. 2008;56:533–538.
24.		Davis D, Galbraith R, American College of Chest Physicians Health and Science Policy Committee. Continuing medical education effect on practice performance: effectiveness of continuing medical education: American College of Chest Physicians Evidence-Based Educational Guidelines. Chest. 2009;135(3 Suppl):42S–48S.
25.		Stewart A, Inglis G, Jardine L, Koorts P, Davies MW. A randomised controlled trial of blended learning to improve the newborn examination skills of medical students. Arch Dis Child Fetal Neonatal Ed. 2013;98(2):F141–F144.
26.		Connell SE, Yates P, Barrett L. Understanding the optimal learning environment in palliative care. Nurse Educ Today. 2011;31(5):472–476.
27.		Curran VR, Sharpe D, Forristall J, Flynn K. Attitudes of health sciences students towards interprofessional teamwork and education. Learn Health Soc Care. 2008;7(3):146–156.
28.		Milanese SF, Grimmer-Somers K, Souvlis T, Innes-Walker K, Chipchase LS. Is a blended learning approach effective for learning in allied health clinicians? Phys Ther Rev. 2014;19(2):86–93.
29.		Browning DM, Solomon MZ, Initiative for Pediatric Palliative Care (IPPC) Investigator Team. The initiative for pediatric palliative care: an interdisciplinary educational approach for healthcare professionals. J Pediatr Nurs. 2005;20(5):326–334.
30.		Mieure KD, Vincent WR, Cox MR, Jones MD. A high-fidelity simulation mannequin to introduce pharmacy students to advanced cardiovascular life support. Am J Pharm Educ. 2010;74(2):22–27.
31.		Renton K, Quinton H, Mayer AT. Educational impact of paediatric palliative simulation study days. BMJ Support Palliat Care. 2017;7(1):88–93.
32.		American Nurses Credentialing Center’s Commission on Accreditation. The importance of evaluating the impact of continuing nurse education on outcomes: professional nursing practice and patient care; 2014. Available from: http://web.archive.org/web/20171030200039/http://www.nursecredentialing.org:80/Accreditation/ResourcesServices/Evaluating-the-Impact-CNE-Outcomes.pdf. Accessed November 29, 2018.
33.		Czurylo K, Gattuso M, Epsom R, Ryan C, Stark B. Continuing education outcomes related to pain management practice. J Contin Educ Nurs. 1999;30(2):84–87.
34.		Guardini I, Talamini R, Lirutti M, Palese A. The effectiveness of continuing education in postoperative pain management: results from a follow-up study. J Contin Educ Nurs. 2008;39(6):281–288.
35.		Schilling K, Applegate R. Best methods for evaluating educational impact: a comparison of the efficacy of commonly used measures of library instruction. J Med Libr Assoc. 2012;100(4):258–269.
36.		Schiekirka S, Feufel MA, Herrmann-Lingen C, Raupach T. Evaluation in medical education: A topical review of target parameters, data collection tools and confounding factors. Ger Med Sci. 2015; 13:1–19.
37.		Ellis LB. Evaluating the effects of continuing nurse education on practice: Researching for impact. NT Research. 1996;1(4):296–305.
38.		Morrison J. Evaluation. BMJ. 2003:326–385.

Supplementary materials

Pre- and post-education survey content

Demographic information collected in the pre-education survey included

• Participant – name, organization, postcode, and occupational group.
• Experience – percentage of week involved providing palliative care directly to patients/families, number of children cared for in the past year who have died, and previous involvement in the care of a child with a life-limiting condition.
• Previous education – previous education days attended led by the Pediatric Palliative Care Service in that state.

Education assessment

• While preparing for this education session, do you feel that you are confident in managing the new referral of a child with palliative care needs to your service?
• While preparing for this education session, do you feel that you are confident in the management of symptoms that a child with a serious life-limiting condition may experience? – pain, nausea, dyspnea, seizures, fear or anxiety
• While preparing for this education session, how confident do you feel in helping a family prepare for the death of their child?
• While preparing for this education session how confident do you feel giving medications, including setting up a subcutaneous delivery system?
• While preparing for this education session do you feel that you have knowledge of resources such as agencies that can assist you in providing palliative care in your area?

The post-education survey repeated these questions preceded by the phrase “since attending the education session” or “after completing the education session…”.

Education topics

The pre- and post-survey forms included open text responses about the topics that participants hoped to hear and topics that they would like to be covered in the future. The questions were as follows.

• Pre-survey – are there any other topics you would like to see covered in this session?
• Post-survey – are there any other topics which you would like to see covered in sessions in the future?