Team Functioning And Beliefs About Team Effectiveness In Inter-Professional Teams: Questionnaire Development And Validation
Received 6 June 2019
Accepted for publication 18 September 2019
Published 4 October 2019 Volume 2019:12 Pages 827—839
Checked for plagiarism Yes
Review by Single anonymous peer review
Peer reviewer comments 2
Editor who approved publication: Dr Scott Fraser
Kelley Kilpatrick,1,2 Lysane Paquette,3 Marissa Bird,4 Mira Jabbour,2 Nancy Carter,4 Éric Tchouaket5
1Susan E. French Chair in Nursing Research and Innovative Practice, Ingram School of Nursing, McGill University, Montreal, Quebec, Canada; 2Centre intégré universitaire de santé et de services sociaux de l’Est-de-l’Île-de-Montréal, Maisonneuve-Rosemont Hospital Site, Montréal, Quebec, Canada; 3Faculty of Nursing, Université de Montreal, Montréal, Quebec, Canada; 4School of Nursing, McMaster University, Hamilton, Ontario, Canada; 5Department of Nursing, Université du Québec en Outaouais (UQO), Saint-Jérôme, Quebec, Canada
Correspondence: Kelley Kilpatrick
Ingram School of Nursing, McGill University, 680 Sherbrooke West, 18th Floor, Office 1811, Montreal, Quebec H3A 2M7, Canada
Tel +1 514 398 3416
Email [email protected]
Purpose: Few validated instruments are available to measure team functioning in acute and primary care teams. To address this, we developed a questionnaire measuring healthcare provider perceptions of team effectiveness (Provider-PTE) and assessed its psychometric properties.
Patients and methods: Empirical evidence and a conceptual model were used for item generation. The 41-item self-completed questionnaire was developed. A cross-sectional survey of healthcare providers (n=283) across a range of settings was performed. Psychometric properties were assessed for French and English language questionnaires using Cronbach alpha (α) for reliability, the feedback form for face validity, expert opinion for content validity, and the known-group technique for construct validity. Responsiveness was examined by comparing scores in high and low functioning teams.
Results: The mean time needed to complete the questionnaire was less than 9 mins. Respondents were typically female (84%), and employed full time (80%) in urban settings (82%). Cronbach α values were as follows: Team Processes = 0.88; PTE-Overall = 0.91; Outcomes = 0.72. Significant differences were found by professional group (p = 0.017), length of time in the team (p = 0.025), and presence of nurse practitioners. Responses to Outcomes varied by employment status (p = 0.017). Differences were identified in high and low functioning teams (p<0.001). Feedback indicated that two questions related to team meetings needed to be added.
Conclusion: The study produced evidence of validity for English and French language Provider-PTE questionnaires. The revised 43-item instrument represents an important contribution by providing a validated questionnaire to measure team functioning across a range of settings that is consistent with a conceptual framework.
Keywords: acute care, inter-professional, psychometric assessment, primary care, questionnaire, validation study
Inter-professional healthcare teams unite providers from different professions (eg, nurse, nurse practitioner, pharmacist, physician) to provide patient care.1 Inter-professional teams represent a safe and effective strategy to deliver care to patients and families because each professional contributes specific knowledge and skills.1,2 These teams were introduced in hospitals over 100 years ago but are relatively new in primary care.3 Team performance has been examined using system-level indicators including access to care, wait times, and screening rates.4,5 Yet, to better understand patient outcomes, it is essential to capture team-level processes.
Processes are defined as interactions that occur at the level of the patient and care provider.6 Team processes are the mechanisms that explain how actions unfold over time in response to what occurs in the surrounding context.7,8 Researchers have often compared processes to a black box where one can observe inputs and outputs without clearly understanding what is happening in between.8–11 Processes including teamwork, communication and patient engagement are cornerstones in the delivery of effective care.12–14 In primary care, Hofhuis et al15 found that team performance was influenced by how frequently team members interacted with each other. However, these authors were unable to measure team-level variance.
Studies have identified that provider judgements or beliefs about team functioning and effectiveness are influenced by processes that include decision-making, communication, cohesion, care coordination, problem-solving, a focus on the needs of patients and families, as well as role clarity and trust among team members.16,17 The measurement of these team processes has been a critical challenge because of the financial and human resources required and the lack of validated instruments. Thus, important gaps in knowledge related to team processes and team functioning remain.18
Greater understanding of how healthcare teams function is crucial given the negative consequences of poor team functioning. In the United States, miscommunication was identified as a key issue following a review of more than 23,000 malpractice claims in 4 years.19 The cost of the incidents was estimated at $1.7 billion US and 2000 lives were lost.19 Similar issues have been identified in Canada, where 37% of adverse events were deemed to be preventable with improved teamwork and communication.20
A systematic review of the experiences of health professionals regarding teamwork and collaboration in primary care found that key attributes of teamwork focused on user needs, interdependent actions, negotiations between professionals, decision-making, mutual respect, trust among team members, and a recognition of the role and work of providers in different professional roles.13 These authors identified that providers in primary care faced enormous challenges to promote teamwork and collaboration. In primary care, Beaulieu et al21 examined organizational structures associated with the delivery of high-quality care and found that provider views of the team’s ability to innovate were moderately associated with outcomes. In acute care, Sidani and Doran22 identified positive associations between care coordination, counselling and education provided by nurse practitioners and patient outcomes related to symptom resolution, improved functional status and satisfaction with care.
Recent systematic reviews have examined the characteristics of instruments designed to measure team functioning and team effectiveness. Important gaps were identified,23,24 including the lack of inclusion of key dimensions of team functioning consistent with a conceptual framework and the need for patients and families as healthcare team members. Additionally, instruments have focused on specific clinical areas such as acute care or primary care but have not been validated across different settings. Validated questionnaires are needed to examine the inter-professional team members’ views of team processes in different settings.
The study aimed to 1) develop a Provider-Perceptions of Team Effectiveness questionnaire (Provider-PTE) to examine processes in healthcare teams from the perspectives of inter-professional team members and to 2) assess the psychometric properties (ie, reliability, face validity, content validity, construct validity) of the Provider-PTE questionnaire.
The study was guided by a conceptual framework developed to support decisions and research related to nurse practitioner roles.25 Nurse practitioners often work in teams that include members from more than one professional group, and the term inter-professional is used in the framework and the current study. The framework includes three central process dimensions: role enactment, boundary work, and perceptions of team effectiveness. Of particular importance to this study is the Perceptions of Team Effectiveness dimension. Key processes in effective teams include improved communication, involvement in decision-making, cohesion, care coordination, problem-solving, and a focus on patient and family needs.16,25,26
A cross-sectional survey was undertaken. A self-administered questionnaire represents a cost-effective strategy to gather information from a large number of participants in distinct geographic locations.27
We recruited participants using a convenience sample. We aimed to recruit 250 care providers to assess the psychometric qualities of the instrument and identify differences in high and low functioning teams. Our sample size allowed us to detect medium correlations and a 95% confidence interval half-width of 0.1 with 80% statistical power and a 5% alpha level.28,29 We included in-hospital or specialized ambulatory care in our definition of acute care.30 Primary care was defined as comprehensive healthcare services for common health concerns at the point of entry to the healthcare system.31
Following Research Ethics Committee approval by the Comité d’éthique de la recherche du CIUSSS de l’Est-de-l’Île-de-Montréal (Project no 2018-1168, CIUSSS_EMTL_282), data were collected from December 2017 to June 2018. The survey included an information sheet, consent form, a feedback form and a questionnaire. Participants were under no obligation to participate in the study. They were advised that they could withdraw at any time. Online and paper copies of the documents were available to participants to enhance response rates.32 All documents were available in English and French.
To address Objective 1, the questionnaire was developed using empirical evidence from multiple case studies of processes in inter-professional teams16,33–35 and a review of the literature describing processes in teams.36
Several steps were undertaken at this stage (Figure 1). To begin, items were generated using examples and descriptions of processes and actions identified by interprofessional team members in cardiac surgery and long-term care.16,33–35 (Figure 1). An initial list of 69 items was generated. Draft items were reviewed by members of the research team to ensure item clarity and legibility. Researchers (n=8) who specialized in questionnaire development and advanced practice nursing research reviewed the instrument. They made suggestions to clarify some items by providing examples and adding descriptions before the start of each section.
Figure 1 Flowchart of the Provider-PTE questionnaire development.
As a class exercise, 30 graduate-level nursing students attending a role development course worked in small groups to generate key contributions of advanced practice nursing to team functioning and patient care using relevant literature and their clinical experience. The contributions were mapped to the items in the questionnaire. All of the students’ ideas mapped to at least one specific item but some contributions were measured by up to five questionnaire items. This allowed the research team to identify redundancies. At the same time, our team was conducting a study to examine the psychometric properties of the Patient-PTE questionnaire.37 The items in both questionnaires were worded in a similar fashion for consistency. Items from the patient questionnaire were examined to determine if any were problematic. From these results, the number of negatively worded items were limited to three in the Provider-PTE questionnaire.
The Provider-PTE questionnaire included 41 items divided into Work Setting, Team Processes, PTE-Overall, and Outcomes presented in Table 1, along with Socio-Demographic Characteristics. More specifically, eight items described the characteristics of the Work Setting (eg, type of healthcare setting, location, caseload, and team size). Team Processes included 14 items. One item specifically measured respondents’ judgements or beliefs about the team’s effectiveness (BE). This score was used to determine high and low functioning teams. Seventeen items measured PTE-Overall and included Team Processes and factors (eg, role clarity) known to influence PTE. Outcomes included seven items (eg, timely care, staff knowledge and skills). Response options ranged from 1 to 6 (ie, strongly disagree, disagree, disagree somewhat, agree somewhat, agree, and strongly agree). A mean value was estimated to obtain a score for each dimension. The average scores ranged from 1 to 6. Improved perceptions of team processes and better team functioning were indicated by higher scores for Team Processes, PTE-Overall, and Outcomes. The socio-demographic characteristics of providers included eight items (eg, age, gender, education, professional group, and employment status). Most response options in this section were categorical. Age and time in the professional role, the organization and the healthcare team were numeric values. One open-ended question gathered any additional comments from respondents. We used single-item measures to measure boundary work (ie, trust), belief about team effectiveness, and cohesion to reduce response burden.38 Two items measured decision-making, coordination, problem-solving, and role clarity. Three items were used to examine communication and a focus on the role of patients and families as team members.
Table 1 Preliminary Items Related To Healthcare Team Processes And Outcomes
The questionnaire was translated from English by a professional translator whose mother tongue is French. It was back-translated by a second translator whose mother tongue is English.39 Strategies to enhance the cross-cultural adaptation of the questionnaires included cognitive interviews with participants to ensure that items were consistently understood in the same manner and pre-testing of the instrument.40 The questionnaires were reviewed by bilingual members of the research team to ensure that the translations maintained semantic and conceptual equivalence with the framework.40,41
Research team members reviewed the draft items. English and French language versions of the questionnaire were pilot-tested with the same advanced practice nursing researchers to identify unclear wording. No revisions were needed following pilot testing. As proposed by Kilpatrick et al,42 the same respondents completed the questionnaire 2 weeks later to determine if their understanding of the items had changed. Participants’ responses and understanding of the questions were unchanged after 2 weeks.
To address Objective 2, cognitive interviewing, a feedback form, and statistical analyses were used to assess the psychometric properties of the questionnaire.
Participants were asked to review instructions, questionnaire layout, procedures to complete French and English language questionnaires, and item and response options for clarity.43 Cognitive interviewing was used to examine the questionnaire with those who completed the pilot test and students in the graduate-level course.
After the questionnaire was completed, a feedback form developed previously was administered.37,42,44 Participants were asked to provide feedback on typical items such as time needed to complete the questionnaire, clarity of instructions; comprehensiveness of the questions related to the topic; readability; and ordering of questions. Participants had the opportunity to provide suggestions to improve the Provider-PTE questionnaire via an open-ended question.
Data were analyzed using the IBM Statistical Package for the Social Sciences (SPSS), version 25.45 Mean, standard deviation (SD), and range were generated using descriptive statistics. Overall Provider-PTE scores (PTE-Overall) and dimensional scores were calculated. Instrument performance was examined using Cronbach alpha (α) to assess reliability, item-total correlation of each item, and assessment of missing data. Responses did not follow a normal distribution (p values < 0.001),46 according to the Shapiro–Wilk test. For variables with two categories, we used the Mann–Whitney U-test, and for variables containing more than two categories, we used the Kruskal–Wallis one-way analysis of variance for nonparametric testing. A Bonferroni correction was applied to correct for multiple comparisons and decrease the likelihood of Type I error. Rank correlation was determined using Spearman’s rho for continuous data.47 Responses to open-ended questions were analyzed using content analysis.48
Internal consistency of the subscales (ie, Team Processes, PTE-Overall, and Outcomes) was assessed using Cronbach alpha. Values ranging from 0.7 to 0.9 were considered acceptable to excellent.49 Because we did not have a sufficiently large number of participants from the same healthcare team, we did not compare the within-group variance to the between-group variance to determine the consistency of responses among participants in the same team.50
Face validity was assessed using the feedback form that examined the Provider-PTE questionnaire, formatting and instructions. Content validity was assessed prior to the questionnaire roll-out by expert groups including researchers and graduate students. Construct validity was examined using the known-group technique.51,52 We compared the scores between specific groups, and hypothesized a priori that no differences in scores would be found between 1) men and women; 2) English and French language respondents; and 3) respondents in urban and rural locations. Differences were expected according to these factors:
- professional group (intra-professional (ie, within the same professional group), inter-professional and managers);
- team size (small (less than 5 members), medium (5–10 members), large (more than 10 members);
- teams with and without nurse practitioners
- employment status (full time and part time).
Nurse practitioners can autonomously evaluate, diagnose, and treat patients for chronic or acute conditions as well as provide advanced nursing care to best meet patient healthcare needs according to the regulated scope of practice in their jurisdiction.53,54 Teams with and without nurse practitioners were identified in our study because the province of Québec Canada planned to implement 2000 nurse practitioners by 2024 to improve care quality and access to care.55
Streiner et al56 define responsiveness as a questionnaire’s ability to detect a meaningful change. However, several authors argue that there is no agreed-upon method to measure responsiveness.56–58 We compared processes in high and low functioning teams and determined a priori that the scores would be different.
In the following section, results including means, correlations, Cronbach α and p values statistically significant at the 5% level or α/n groups for post-hoc analysis are reported by measure and known group.
Overall, 314 participants responded to the survey. Thirty-one questionnaires included only socio-demographic data and were excluded. Thus, 283 questionnaires were included in the analyses. Most questionnaires were completed online (75%, n=211). The mean time to complete the questionnaire was 21 mins. When three outliers were removed (ie, 243 min, 1447 min, and 1753 min), the mean time to complete the questionnaire decreased to 8.9 mins (SD: 8.75).
Socio-demographic characteristics of participants are presented in Table 2. A majority of respondents completed the questionnaire in French (85%, n=241). Most respondents were female (84%, n=216), and a majority worked full time (80%, n=205). The average age of respondents was 39 years of age (SD: 9.1, range 23–67). Providers had worked in their professional role for 13.3 years (ranging from 1 month to 45 years), in their current role for an average of 7.2 years (ranging from 1 month to 45 years), and in their current team for 6.4 years (ranging from 1 month to 45 years).
Table 2 Characteristics Of Respondents
Healthcare team characteristics are presented in Table 3. Respondents worked primarily in large teams (69%) located in an urban setting (82%). Specialties included primary care (56%), oncology (16%), Emergency/Intensive care unit (12%), and medicine (16%). Half of the respondents worked in specialized outpatient settings and in-patient units. Respondents worked in teams who cared for a wide range of patient populations.
Table 3 Characteristics Of The Healthcare Team
Reliability was examined using Cronbach α values for Team Processes, PTE-Overall, and Outcomes. Cronbach α values for Team Processes equaled 0.88; PTE-Overall equaled 0.91, and Outcomes equaled 0.72 (see Table 4).
Table 4 Mean Scores For Boundary Work, Perceptions Of Team Effectiveness, Role Clarity, And Outcomes
Face validity was assessed using responses (n=85) in the feedback form (see Table 5). Most (76%) believed that key questions to understand team functioning were included in the questionnaire and no question was unimportant (87%). Positive assessments were noted for the time needed to complete the questionnaire (96%), instructions (99%), formatting (100%), font size (100%), and logical ordering of questions (96%). Some respondents noted that it would be easier to see the negatively worded items if these expressions (eg, not) were bolded.
Table 5 Responses Related To Face Validity
Content validity was assessed using cognitive interviewing and the responses from the open-ended question in the feedback form (n=85). No question was perceived to be redundant. A small number of respondents found it difficult to select a specific patient group if they worked with several patient populations.
Overall, trust and beliefs about team effectiveness (BE) were ranked highest (mean: 5.2/6±0.8, range 1–6). The dimension PTE-Overall rated highest (mean: 4.8/6 ±0.6, range 1.4–6) followed by Team Processes (mean: 4.7/6 ±0.6, range 1.5–6) and Role Clarity (mean: 4.7/6 ±1.0, range 1.0–6). Outcomes were rated 4.8/6 ±0.6 (range 2–6). The ratings for the variables included in the PTE-Overall dimension ranged from 4.4/6 ±1.1 for problem-solving to 5.2/6± 0.8 for the BE score. The rho coefficients (rs) for the variables included in the PTE-Overall dimension ranged from 0.58 to 0.81 (p value< 0.001) (see Table 4).
Construct validity was examined using known groups (see Table 6). As anticipated, no differences were found between men and women, English and French language respondents, and respondents in urban and non-urban locations. As hypothesized, differences were identified according to professional group (p = 0.017), length of time in the team (p = 0.025), and presence of NPs. We had anticipated differences according to team size but no differences were identified between these groups. Differences were noted in Outcomes according to employment status (p = 0.017).
Table 6 Differences In Team Processes And Outcomes By Known Group Comparisons
Responsiveness was assessed using the scores in high and low functioning teams. Significant differences (p<0.001) were noted in all the scores for all the dimensions (see Table 7).
Table 7 Differences In Processes And Outcomes In High And Low Functioning Teams
The Provider-PTE questionnaire is a self-reported instrument available in English and French that measures perceptions of team functioning and team outcomes. The questionnaire includes provider and team characteristics and three subscales (ie, PTE-Overall, Team Processes, and Outcomes). Psychometric testing (n=283) included construct validity using known groups. As anticipated, differences were identified by professional group, teams with nurse practitioners, and length of time in the team. Reliability was examined using Cronbach α. Subscale values ranged from 0.72 to 0.91. Rho coefficients (rs) for processes included in the Team Processes subscale ranged from 0.59 to 0.81 (p< 0.001), indicating that these processes were highly correlated with beliefs about team effectiveness. Responsiveness was assessed, and differences were noted in high and low functioning teams. The questionnaire fills important gaps identified in systematic reviews of instruments that measure team processes by integrating a conceptual framework and examining inter-professional team functioning across different settings.23,24
More recently, one member of the research team (LP) pilot tested several data collection tools, including the Provider-PTE questionnaire, for her doctoral study examining nursing role enactment in intensive care units (ICUs). Participants (n=8) in the pilot test highlighted that one item needed to be clarified and two items needed to be added to the Provider-PTE questionnaire to better capture effective team functioning in the ICU. The item about patient care goals “The plan of care and patient care objectives are clearly outlined in the patient’s health record” was changed to “Healthcare team members know the goals of patients’ plans of care”. This item was reworded because some respondents indicated that unlicensed care providers who do not have access to the patient’s health record would also need to have an understanding of the goals of care. In addition, participants noted the importance of scheduling regular team meetings to plan patient care. We added two items to measure the conduct of regular team meetings in addition to meetings scheduled for emerging issues “Regular inter-professional team meetings are scheduled to plan patient care” and assess if the team could meet quickly to address emerging issues “The healthcare team meets as needed to address complex care issues that arise”. These changes were consistent with the preliminary findings of a systematic review that our team is conducting to highlight the contributions of team meetings to improved team functioning.59
The current study generated preliminary data of the validity of the Provider-PTE questionnaire. Further testing is required for the revised 43-item questionnaire (see Table 8). The next steps include recruiting team members in large existing teams to examine whether perceptions within each team are less varied than between teams. This is important to determine our ability to aggregate survey findings from individuals to the team and organizational levels.50,60 In addition, repeated measurements with the same teams at different time points are needed to generate further evidence of the instrument’s responsiveness and allow for the determination of a minimally important difference in scores that can be used in different context including research (eg, a randomized control trial) or ongoing quality improvement initiatives.58,61 As proposed by Husebo and Akerjordet,62 measurements can be taken before and after a change in practice is undertaken to examine its impact on team functioning. Thus, it will be possible to tease out factors that are known to influence team functioning including trust, role clarity and the conduct of team meetings.59
Table 8 Revised Items Related To Healthcare Team Processes And Outcomes*
The questionnaire makes an important contribution to the teamwork literature because it was developed to be consistent with a conceptual framework. The instrument was validated in English and French, in two different Canadian provinces, and across a range of clinical settings. We were unable to determine the extent of non-response bias because we did not have information on non-responders.63 Strategies to reduce the risk of response bias included the use of pilot testing and cognitive interviewing.63 Simple response options were used to reduce cognitive load when answering questions, and the questionnaire was kept as short as possible while cohering with the conceptual framework.56 We anticipate that the recruitment of healthcare providers from different professions, working in acute and primary care with different patient populations in two Canadian provinces, will enhance the generalizability of our findings to other inter-professional teams.
Previous studies have focused on teams in specific contexts, including primary care,64,65 and acute care62,66,67 or in specialized clinical areas including mental health68 and medicine.69 The Provider-PTE questionnaire can be administered in intra- and inter-professional teams to examine team functioning. It can be used to assess team functioning before and after changes in skill mix, in different settings from acute to primary care, and in-patient and out-patient settings. Given that higher scores indicate better team functioning, it will be possible to monitor team performance to identify processes that function well in addition to those in need of improvement. Such measurements will provide feedback to decision-makers, team members as well as patients and families.
Our study included a convenience sample. Subsequent studies will need to be conducted using a random sample of providers. We did not recruit a sufficiently large number of providers working in the same team or a large enough sample of managers. Future studies will need to be conducted in larger inter-professional teams in different settings to accrue additional evidence of validity and responsiveness. In addition, we administered the questionnaire to members of one healthcare team without direct patient care activities. Some items were difficult to answer for team members without patient contact, which suggests that survey items would need to be adapted for teams not involved in direct patient care. Finally, we did not use factor analysis to examine construct validity because the proposed subscales had fewer than three items per scale.70 Future research is needed to test the questionnaire’s structure and confirm the proposed subscales.
The study produced evidence of validity for the English and French language versions of the Provider-Perceptions of Team Effectiveness questionnaire. The revised 43-item instrument is a self-completed questionnaire that takes less than 9 mins to complete. The questionnaire measures perceptions of team processes and outcomes of all members of the inter-professional team. Reliability indices were acceptable to excellent for the subscales. Face validity and content validity were examined, while construct validity was assessed using known groups. Responsiveness was examined by comparing high and low functioning teams. The questionnaire can be used to examine how inter-professional teams function in different contexts.
α, alpha; MS, mean square; NPs, nurse practitioners; PTE, perceptions of team effectiveness; rs, rho coefficients; SD, standard deviation.
Availability Of Data
The dataset analysed in the current study is available from the corresponding author on reasonable request.
The authors would like to sincerely thank the staff and managers who took time out of their busy schedules to support the study. The authors would like to acknowledge the support from the McGill Nursing Collaborative for Education and Innovation in Patient and Family Centered Care (Newton Foundation/McGill Faculty of Medicine) and the Writing for Publication Initiative (Graphos/McGill Writing Centre and Marc Ducusin). Partnership funding for the study was received from the Réseau de recherche en interventions en sciences infirmières du Québec(RRISIQ)/Québec Network on Nursing Intervention Research, the Ministère de la santé et des services sociaux du Québec, and Fonds de recherche du Québec-Santé (FRQS). KK and ÉT received junior researcher salary awards from the FRQS. The funding bodies played no role in study development, data collection, data analysis, interpretation or in the writing of the manuscript.
The authors report no conflicts of interest in this work.
1. Dirks JL. Effective strategies for teaching teamwork. Crit Care Nurse. 2019;39(4):40–47. doi:10.4037/ccn2019704
2. Nelson S, Turnbull J, Bainbridge L, et al. Optimizing Scopes of Practice: New Models of Care for a New Health Care System. Ottawa (Ontario): Canadian Academy of Health Sciences; 2014. Available from: http://www.cahs-acss.ca/wp-content/uploads/2014/08/Optimizing-Scopes-of-Practice_REPORT-English.pdf.
3. Gilbert F, Denis JL, Lamothe L, Beaulieu MD, D’Amour D, Goudreau J. Reforming primary healthcare: from public policy to organizational change. JHOM. 2015;29(1):92–110. doi:10.1108/JHOM-12-2012-0237
4. Haj-Ali W, Hutchison B. Establishing a primary care performance measurement framework for Ontario. Healthc Policy. 2017;12(3):66–79.
5. Health Quality Ontario. Quality in primary care: setting foundation for monitoring and reporting in Ontario. 2015. Available from: www.hqontario.ca/Portals/0/Documents/pr/theme-report-quality-in-primary-care-en.pdf.
6. Donabedian A. Evaluating the quality of medical care. Milbank Q. 2005;83(4):691–729. doi:10.1111/j.1468-0009.2005.00397.x
7. Van de Ven AH. Suggestions for studying strategy process: a research note. SMJ. 1992;13:169–188.
8. Ramaswamy R, Reed J, Livesley N, et al. Unpacking the black box of improvement. Int J Qual Health Care. 2018;30(suppl.1):15–19. doi:10.1093/intqhc/mzy009
9. Thomson A, Perry J. Collaboration processes: inside the black box. Publ Admin Rev. 2006;66(1 Special Issue):20–32. doi:10.1111/j.1540-6210.2006.00663.x
10. Walker MJ, Bourke J, Hutchison K. Evidence for personalised medicine: mechanisms, correlation, and new kinds of black box. Theor Med Bioeth. 2019;40(2):103–121. doi:10.1007/s11017-019-09482-z
11. Fraser D. A black box or a comprehensive model. Technometrics. 1968;10(2):219–229. doi:10.1080/00401706.1968.10490557
12. Frankel A, Haraden C, Federico F, Lenoci-Edwards J. A Framework for Safe, Reliable, and Effective Care. White Paper. Cambridge (MA): Institute for Healthcare Improvement and Safe & Reliable Healthcare; 2017. Available from: https://static1.squarespace.com/static/582d3a2c197aeae502998512/t/5896021b46c3c41ea2053c8a/1486225950663/FrameworkSafeReliableEffectiveCareWhitePaper.pdf.
13. Sangaleti C, Schveitzer MC, Peduzzi M, Zoboli E, Soares CB. Experiences and shared meaning of teamwork and interprofessional collaboration among health care professionals in primary health care settings: a systematic review. JBI Database System Rev Implement Rep. 2017;15(11):2723–2788. doi:10.11124/JBISRIR-2016-003016
14. Schottenfeld L, Petersen D, Peikes D, et al. Creating Patient-Centered Team-Based Primary Care. White paper. AHRQ Pub No 16-0002-EF. Rockville (MD): Agency for Healthcare Research and Quality; 2016. Available from: https://pcmh.ahrq.gov/page/creating-patient-centered-team-based-primary-care.
15. Hofhuis J, Mensen M, ten Den L, et al. Does functional diversity increase effectiveness of community care teams? The moderating role of shared vision, interaction frequency, and team reflexivity. J Appl Soc Psychol. 2018;48:535–548. doi:10.1111/jasp.12533
16. Kilpatrick K. How do nurse practitioners in acute care affect perceptions of team effectiveness? J Clin Nurs. 2013;22:2632–2647. doi:10.1111/jocn.12198
17. Kilpatrick K, Jabbour M, Tchouaket E, Acorn M, Donald F, Hains S. Implementing primary healthcare nurse practitioners in long term care teams: a qualitative descriptive study. J Adv Nurs. 2019;75(6):1306–1315. doi:10.1111/jan.13962
18. Ryan BL, Brown JB, Oates M, Markle EK, Clark N, Selkirk K. Taking the pulse of team functioning in interprofessional primary health care teams. Can Fam Physician. 2016;62(9):e566–e567.
19. CRICO Strategies. Malpractice risks in communications failures- 2015 annual benchmarking report. 2015. Available from: https://www.rmf.harvard.edu/~/media/0A5FF3ED1C8B40CFAF178BB965488FA9.ashx.
20. Canadian Patient Safety Institute. Canadian framework for teamwork and communication: literature review, needs assessment, evaluation of training tools and expert consultations. 2011. Available from: http://www.patientsafetyinstitute.ca/en/toolsResources/teamworkCommunication/Documents/Canadian%20Framework%20for%20Teamwork%20and%20Communications.pdf.
21. Beaulieu MD, Haggerty J, Tousignant P, et al. Characteristics of primary care practices associated with high quality of care. Can Med Assoc J. 2013;185(12):E590–E596. doi:10.1503/cmaj.121802
22. Sidani S, Doran D. Relationships between processes and outcomes of nurse practitioners in acute care: an exploration. J Nurs Care Qual. 2010;25(1):31–38. doi:10.1097/NCQ.0b013e3181b1f41e
23. Kash BA, Cheon O, Halzack NM, Miller TR. Measuring team effectiveness in the health care setting: an inventory of survey tools. Health Serv Insights. 2018;11:1–18.
24. Valentine MA, Nembhard IM, Edmondson AC. Measuring teamwork in health care settings: a review of survey instruments. Med Care. 2015;53(4):e16–e30.
25. Kilpatrick K, Lavoie-Tremblay M, Lamothe L, Ritchie JA, Doran D. Conceptual framework of acute care nurse practitioner role enactment, boundary work, and perceptions of team effectiveness. J Adv Nurs. 2013;69(1):205–217. doi:10.1111/j.1365-2648.2012.06046.x
26. Kilpatrick K, Jabbour M, Fortin C. Processes in healthcare teams that include nurse practitioners: what do patients and families perceive to be effective? J Clin Nurs. 2016;25(5–6):619–630. doi:10.1111/jocn.13085
27. Babbie E. The Practice of Social Research.
28. Moinester M, Gottfried R. Sample size estimation for correlations with pre-specified confidence interval. TQMP. 2014;10(2):124–130. doi:10.20982/tqmp.10.2.p0124
29. Prajapati B, Dunne M, Armstrong R. Sample Size Estimation and Statistical Power Analyses; OT. 2010;2010(July):1–9.
30. Kilpatrick K, Harbman P, Carter N, et al. The acute care nurse practitioner role in Canada. Can J Nurs Leadersh. 2010;23(Special Issue):114–139. doi:10.12927/cjnl.2010.22272
31. Valaitis R, Martin-Misener R, Wong ST, MacDonald M, Meagher-Stewart D, Austin P. Strengthening primary care through public health and primary care collaboration team. Methods, strategies and technologies used to conduct a scoping literature review of collaboration between primary care and public health. Prim Health Care Res Dev. 2012;13(3):219–236. doi:10.1017/S1463423611000594
32. Kroth PJ, McPherson L, Leverence R, et al. Combining web-based and mail surveys improves response rates: a PBRN study from PRIME Net. Ann Fam Med. 2009;7(3):245–248. doi:10.1370/afm.944
33. Kilpatrick K, Lavoie-Tremblay M, Ritchie JA, Doran D, Lamothe L. Boundary work and the introduction of acute care nurse practitioners in healthcare teams. J Adv Nurs. 2012;68(7):1504–1515. doi:10.1111/j.1365-2648.2011.05895.x
34. Kilpatrick K, Lavoie-Tremblay M, Ritchie JA, Lamothe L, Doran D, Rochefort C. The role components and role enactment of cardiology acute care nurse practitioners. Int J Nurs Stud. 2012;49:850–862. doi:10.1016/j.ijnurstu.2012.01.011
35. Kilpatrick K, Hains S, Tchouaket E, Jabbour M. Implantation du rôle d’infirmière praticienne spécialisée en soins de première ligne en centres d’hébergement et de soins de longue durée au Québec: Projet Vitrine. Ministère de la Santé et des Services Sociaux; 2017.
36. Kilpatrick K, Lavoie-Tremblay M, Ritchie JA, Lamothe L. Advanced practice nursing, health care teams, and perceptions of team effectiveness. J Trauma Nurs. 2014;21(6):291–299. Reprinting with permission from The Health Care Manager, 30(3), 215–226. doi:10.1097/JTN.0000000000000090
37. Kilpatrick K, Tchouaket E, Paquette L, et al. Measuring patient and family perceptions of processes in healthcare teams: questionnaire development and psychometric evaluation. BMC Health Serv Res. 2019;19(9):1–16. doi:10.1186/s12913-018-3827-x
38. Fisher GG, Matthews RA, Gibbons AM. Developing and investigating the use of single-item measures in organizational research. J Occup Health Psychol. 2016;21(1):3–23. doi:10.1037/a0039139
39. Ballangrud R, Husebo SE, Hall-Lord ML. Cross-cultural validation and psychometric testing of the Norwegian version of the TeamSTEPPS(R) teamwork perceptions questionnaire. BMC Health Serv Res. 2017;17(1):799. doi:10.1186/s12913-017-2733-y
40. Sidani S, Guruge S, Miranda J, Ford-Gilboe M, Varcoe C. Cultural adaptation and translation of measures: an integrated method. Res Nurs Health. 2010;33(2):133–143. doi:10.1002/nur.20364
41. Guillemin F, Bombardier C, Beaton D. Cross-cultural adaptation of health-related quality of life measures: literature review and proposed guidelines. J Clin Epidemiol. 1993;46(12):1417–1432. doi:10.1016/0895-4356(93)90142-n
42. Kilpatrick K, DiCenso A, Bryant-Lukosius D, Ritchie JA, Martin-Misener R, Carter N. Practice patterns and perceived impact of clinical nurse specialist roles in Canada: results of a national survey. Int J Nurs Stud. 2013;50(11):1524–1536. doi:10.1016/j.ijnurstu.2013.03.005
43. Kelley K, Clark B, Brown V, Sitzia J. Good practice in the conduct and reporting of survey research. Int J Qual Health Care. 2003;15(3):261–266. doi:10.1093/intqhc/mzg031
44. Bryant-Lukosius D, Green E, Fitch M, et al. A survey of oncology advanced practice nurses in Ontario: profile and predictors of job satisfaction. Can J Nurs Leadersh. 2007;20(2):50–68.
45. IBM. SPSS Statistics Version 25. New York; 2017. Available from: http://www.ibm.com/contact/us/en/?lnk=flg-cont-usen?lnk=flg.
46. Kutner MH, Nachtsheim CJ, Neter J, Li W. Applied Linear Statistical Models.
47. Kellar SP, Kelvin EA. Munro’s Statistical Methods for Healthcare Research. Philadelphia (PA): Lippincott, Williams & Wilkins; 2013.
48. Elo S, Kyngäs H. The qualitative content analysis process. J Adv Nurs. 2008;62(1):107–115. doi:10.1111/j.1365-2648.2007.04569.x
49. Tabachnick BG, Fidell LS. Using Multivariate Statistics.
50. Wolak ME, Fairbairn DJ, Paulsen YR. Guidelines for estimating repeatability. Methods Ecol Evol. 2012;3:129–137. doi:10.1111/j.2041-210X.2011.00125.x
51. Davidson M. Known-Groups Validity. Encyclopedia of quality of life and well-being research. Dordrecht: Springer; 2014:3481–3482.
52. Polit D, Beck C. Essentials of Nursing Research: Appraising Evidence for Nursing Practice.
53. Poghosyan L, Liu J, Norful AA. Nurse practitioners as primary care providers with their own patient panels and organizational structures: a cross-sectional study. Int J Nurs Stud. 2017;74:1–7. doi:10.1016/j.ijnurstu.2017.05.004
54. Carney M. Regulation of advanced nurse practice: its existence and regulatory dimensions from an international perspective. J Nurs Manag. 2016;24(1):105–114. doi:10.1111/jonm.12278
55. Radio-Canada. Infirmières praticiennes spécialisées: québec ajoute 25 millions au 1,4 milliard sur 10 ans. 2017. Available from: https://ici.radio-canada.ca/breve/85738/infirmieres-praticiennes-specialisees-quebec.
56. Streiner DL, Norman GR, Cairney J. Health Measurement Scales: A Practical Guide to Their Development and Use.
57. Revicki D, Hays RD, Cella D, Sloan J. Recommended methods for determining responsiveness and minimally important differences for patient-reported outcomes. J Clin Epidemiol. 2008;61(2):102–109. doi:10.1016/j.jclinepi.2007.03.012
58. Guyatt G, Deyo A, Charlson M, Levine M, Mitchell A. Responsiveness and validity in health status measurement: a clarification. J Clin Epidemiol. 1989;42(5):403–408. doi:10.1016/0895-4356(89)90128-5
59. Paquette L, Kilpatrick K, Jabbour M, et al.
60. Verran JA, Gerber RM, Milton DA. Data aggregation: criteria for psychometric evaluation. Res Nurs Health. 1995;18(1):77–80.
61. Martel J, Dugas C, Lafond D, Descarreaux M. Validation of the French version of the Bournemouth Questionnaire. J Can Chiropr Assoc. 2009;53(2):102–120.
62. Husebo SE, Akerjordet K. Quantitative systematic review of multi-professional teamwork and leadership training to optimize patient outcomes in acute hospital settings. J Adv Nurs. 2016;72(12):2980–3000. doi:10.1111/jan.13035
63. Sedgwick P. Questionnaire surveys: sources of bias. BMJ. 2013;347(f5265):1–2.
64. Song H, Chien AT, Fisher J, et al. Development and validation of the primary care team dynamics survey. Health Serv Res. 2015;50(3):897–921. doi:10.1111/1475-6773.12257
65. Song H, Ryan M, Tendulkar S, et al. Team dynamics, clinical work satisfaction, and patient care coordination between primary care providers: a mixed methods study. Health Care Manage Rev. 2017;42(1):28–41. doi:10.1097/HMR.0000000000000091
66. Weld LR, Stringer MT, Ebertowski JS, et al. TeamSTEPPS improves operating room efficiency and patient safety. Am J Med Qual. 2016;31(5):408–414. doi:10.1177/1062860615583671
67. Natafgi N, Zhu X, Baloh J, Vellinga K, Vaughn T, Ward MM. Critical access hospital use of TeamSTEPPS to implement shift-change handoff communication. J Nurs Care Qual. 2017;32(1):77–86. doi:10.1097/NCQ.0000000000000203
68. El Ansari W, Lyubovnikova J, Middleton H, Dawson JF, Naylor PB, West MA. Development and psychometric evaluation of a new team effectiveness scale for all types of community adult mental health teams: a mixed-methods approach. Health Soc Care Community. 2016;24(3):309–320. doi:10.1111/hsc.12203
69. Havyer RD, Wingo MT, Comfere NI, et al. Teamwork assessment in internal medicine: a systematic review of validity evidence and outcomes. J Gen Intern Med. 2014;29(6):894–910. doi:10.1007/s11606-013-2686-8
70. Costello AN, Osborne JW. Best practices in exploratory factor analysis: four recommendations for getting the most from your analysis. PARE. 2005;10(7):1–8.
This work is published and licensed by Dove Medical Press Limited. The full terms of this license are available at https://www.dovepress.com/terms.php and incorporate the Creative Commons Attribution - Non Commercial (unported, v3.0) License. By accessing the work you hereby accept the Terms. Non-commercial uses of the work are permitted without any further permission from Dove Medical Press Limited, provided the work is properly attributed. For permission for commercial use of this work, please see paragraphs 4.2 and 5 of our Terms.