Analysis of Self- and 360-Evaluation Scores of the Professionalism Intelligence Model Within an Academic Otolaryngology-Head and Neck Surgery Department

Introduction

Due to numerous challenges, regulatory issues, and rapid changes facing the delivery of medical care, health care institutions have seen a remarkable growth of leadership development programs with a particular focus on physicians and nurses.^1–4 These initiatives improved health care quality and cost by reducing clinical errors, mortality rates and lengths of hospitalization.^5–7 However, most of these programs speak of leadership development while they refer to individual leader development. The terms “leader” and “leadership” development are often used interchangeably with no distinction between them.⁸ In fact, leadership is a complex phenomenon that encompasses the interactions between the leader and the social and organizational environment.^9,10 Additionally, the authors of a systematic review on leadership development programs for physicians noticed a lack of programs that integrate non-physicians and physicians professionals,³ all of whom are involved in patients’ interaction in the healthcare setting.

To address this issue, a leadership course named Leadership Lived Out (LLO) was introduced in the Department of Head and Neck Surgery & Communication Sciences at Duke University Medical Center. This yearlong program has a patient centered approach and is tailored to all staff members including physicians, clinic and operating room nurses, trainees, administrators, and staff. As part of the program, the cohort of 7–10 participants per year completed a self and 360 assessment called the Professionalism Quotient Inventory (PQ-I) based on the Professionalism Intelligence Model (PIM) and received personal coaching.¹¹ The PQ-I assesses behaviors related to the three domains of Cognitive intelligence, Emotional intelligence, and Leadership intelligence. Our primary hypothesis is that there exists significant association between gaps in self-score indicating self-awareness and 360-score or other’s perception, for each PIM domain as it relates to healthcare team roles, gender, and generational group.

Methods

IRB exemption was declared from Duke University Medical Center Institutional Review Board since it met the criteria of exemption 2ii of the Revised Common Rule. Participants who completed the PQ-I were included in this study.

Data Collection and Management

The PQ-I involves participants completing questions about themselves based on their perception. The PQ-I 360 is a multi-rater measure about the participant completed by those that they report to (manager), those considered at the same level (peers), and those that they supervise (direct reports). All PQ-I 360 raters are anonymous and pooled by group (manager, peers, direct reports). Suggested raters are provided by the participant. The final report combines a self-evaluation with that of the raters, providing comparison of the overall Professionalism Intelligence capabilities. The PQ-I questionnaire was completed by each participant approximately 3 months into the program. The scores were measured in 3 domains of Cognitive Intelligence, Emotional Intelligence, and Leadership Intelligence. Each of these domains had 5 subdomains which are shown in Figure 1.

The primary outcomes were gaps, defined as the difference between self-score that indicate self-awareness with that of the 360 scores, for each evaluator who provided a score to the participant among the three domains.

Statistical Analysis

The three variables of interest included gender (male vs female), age based on two generational groups based on year of birth: generation Y (1982–2001) or older generations (prior to 1982), and physician status (physician vs non-physician).

Participant’s characteristics were summarized with mean (standard deviation, SD) for age and with frequency (percentage) for gender and physician status. Participant self-scores were presented with mean (SD). Analyses were performed separately for each domain and subdomain. Differences in mean self-scores between participant genders, generations, and physician status were modeled using linear regression. Two types of linear mixed models were fit for gaps in scores with a random intercept to account for the correlation in the same participant. The first type was a single model that contained evaluator role (peer, direct report, or manager/supervisor) as a main effect which allowed us to estimate the gaps in scores for each evaluator role. The second type was a set of models that included evaluator-role and one of the participant characteristics (age, gender, or physician status), and their cross-products. Contrast statements were used to estimate the difference in gaps between levels of participant characteristics for each evaluator role. Coefficient estimates, β, and their 95% confidence intervals (CI), were reported for each model. All analyses were performed in R 3.6.3 (R Core Team, 2019).¹² Mixed-models were fit using the lme4 package.¹³

Results

Demographics

A total of 53 participants were in the original data set. Among these, 3 participants did not receive any feedback from their evaluators and were excluded from the analyses. The final analysis dataset consisted of 50 participants and 394 evaluator members. The average age was 40.9 (SD 9.7) and half of the participants were females (50%). Approximately one third of the participants were physicians (N=18). Among these physicians, 38.9% (N=7) were attending and 61.1% (N=11) were residents (Table 1). Four of the 50 participants included in the analysis were in the baby boomer generation and were added to the 26 participants in Generation X to form the Older Generations group.

Table 1 Participants Characteristics

Self-Scores

Self-scores for the three domains are reported by gender, generation, and physician status (Table 2). The self-scores in the domains are similar between gender, generation, and physician status, except for leadership intelligence between physicians and non-physicians. On average, physicians scored themselves lower than non-physician participants (80.3 (SD 7.4) vs 75.7 (SD 5.6); = −4.57; 95% CI: −8.62, −0.53). The analysis of leadership intelligence subdomains showed that both “embodiment” and “empowerment” had an inverse association with physician status. While cognitive intelligence and emotional intelligence did not yield any significant association with physician status, some of their subdomains did. In fact, physicians scored themselves significantly lower compared to non-physicians when assessing their “interpersonal relationships”, “empathy”, and “focused thinking” (Figure 2).

Table 2 Domain Self-Scores by Gender, Generation, and Physician Status

Evaluator Score vs Self Score

The average gap between self-score and evaluator-score was calculated as (self-score) – (evaluator-score) for each type of work evaluator member (Table 3). All coefficient values came back negative, indicating that participants tended to rate themselves lower than their evaluators in all domains, regardless of the role of the rater. Direct reports rate the participants the highest on average, and managers/supervisors the lowest, although still significantly higher than the participants rate themselves (Table 3).

Table 3 β (95% Confidence Interval) for Domain Score Gaps by Evaluator Role

Furthermore, the average gap between self-score and evaluator-score for each type of work evaluator member was estimated separately for each gender, generation group, and physician status. The same rating pattern was observed; participants tended to score themselves lower compared to their evaluator members even when the gaps were broken down by their characteristics. However, there were significant associations between age groups and cognitive intelligence in peers and managers/supervisors (Table 4). Participants in the older generations group on average rated themselves 8.76 (95% CI: −11.58, −5.94) points lower than their peers, whereas participants from Generation Y rate themselves only 3.59 (95% CI: −7.11, −0.08) points lower than their peers (β = 5.16; 95% CI: 0.66, 9.67; p = 0.025). Similarly, participants from the Older Generations on average rate themselves 6.49 (95% CI: −9.80, −3.18) points significantly lower than their managers/supervisors, whereas participants from Generation Y only rate themselves 1.18 (95% CI: −5.06, 2.70) points lower than their managers (β = 5.31; 95% CI: 0.21, 10.41; p = 0.042).

Table 4 β (95% Confidence Interval) for Domain Score Gaps by Participant Age and Evaluator Role

Discussion

Self-awareness and understanding the perception among others that comprise a healthcare team about oneself is important to leadership and professional development.^14,15 This is the first report of an analysis of 360 evaluations from a leadership program that includes all roles in a healthcare team (ie LLO). The results have several observations that deserve consideration in the context of other research findings.

We report that all participants had a lower perception of their leadership, cognitive, and emotional intelligence compared to their evaluator’s perception. Prior research has generally shown that under-estimators of themselves are rated more favorably on their leadership performance.¹⁶ An explanation for the lower self-perception seen is that participants’ self-ratings of leadership capabilities decrease from baseline to the end of a program as a result of developing a deeper understanding of leadership. In addition, the challenges faced by the participants when applying their learnings to the workplace could explain low self-ratings.^17,18

While gender did not yield any significant association with self-scores in this study, Vecchio et al reported that men have a tendency to over-estimate their effectiveness as leaders compared to women.¹⁹ Interestingly, physicians scored themselves significantly lower than non-physicians when assessing their leadership intelligence particularly embodiment and empowerment. Alternatively, the lower rating may be a recognition of needed leadership development. Sixty-one percent of the physicians in this study were residents. Bent et al.found that most residents in an Otolaryngology department do not consider themselves as good leaders.²⁰ This highlights the importance of incorporating leadership programs into resident’s curriculum.

Emotional intelligence is a fundamental component of effective practice and is generating increased interest in the field of health care.²¹ From hospital administrators to physicians and nurses, collaboration is required not only to improve cost effectiveness of practice but also to ensure patient compliance and satisfaction.²² This characteristic is particularly crucial in physicians who are called for breaking down barrier of communications with patients, in favor of a more empathic approach. McKinley et al.concluded that resident physicians demonstrated a global emotional intelligence similar to that of the general population and called for targeted educational interventions that emphasize different aspects of emotional intelligence.²³

Groups of people born in different time periods develop divergent cultural values and traits, which also manifest in the workplace. Institutions must harness the attributes of each generation to meet the demands of their organization and create effective teams.²⁴ There is a paucity of information regarding the generational differences in healthcare except a number of studies assessing those differences in the nursing workforce.^24,25 The Millennials or Generation Y value teamwork with credits and accomplishments being assigned equally to all members of team. Generation Y emphasizes the notion of belonging to a group that employers who neglect this feature find little success in motivating them.²⁶ Older Generations are more interested in a healthy work/life balance and in new skills and technologies.²⁵

The Older Generations group was associated with a lower self-rating when relating to work evaluator feedback. In contrast, prior studies showed that older managers, as compared to younger managers, tended to over-rate their performance in relation to ratings provided by their supervisors.^19,27–29 Given that experience comes with age, we could assume that those with longer tenure will provide higher self-ratings relative to others’ ratings which is not what was found in our data. This phenomenon may be explained by the Dunning – Kruger effect which is a cognitive bias described by social psychologists David Dunning and Justin Kruger.³⁰ They conducted a series of studies showing that less experienced people tend to over-estimate themselves by being less well calibrated in their ability to judge their performance. This shows a lack of self-awareness since there is a gap between their own perceptions of their abilities and their actual performance. While highly experienced people tend to under-estimate themselves by having a misperception of their high ability, this may be a reflection sign of leadership maturity in that their standards and expectations are placed higher for themselves than what others perceive.

This is the first study to assess the PQ-I within an academic health care department. The use of a 360-degree feedback proved to be an effective method for initiating growth and development of leadership acumen.^31,32 Some authors argue that leadership effectiveness may improve by 60% in programs that rely on 360-degree feedback and coaching.³² However, this improvement is dependent on the presence of post-feedback support through leadership development activities.³³ LLO incorporated both 360 evaluations and also personal coaching focused on these results. We believe that the results shown in our study are unique in how a leadership program not only gives insights into self and other perception of defined leadership virtues in both physicians and non-physicians with a healthcare team. In addition, these findings helped creating post-feedback support for all participants in LLO course. The results of this study should further be considered within the context of potential limitations to this work. Although our sample size was adequate to detect statistically significant differences between groups, a larger sample size may provide greater stability of the estimates and the opportunity to assess other variables like the academic level and the cultural background. We continue to offer the program and intend to further assess these findings. Moreover, the work evaluators were randomly selected from a list of names presented by the participants themselves, leading to selection bias. Finally, raters might be biased towards higher scores to avoid perceived lack of anonymity.

Conclusions

Effective leadership in healthcare requires cognitive, emotional, and leadership intelligence as based on the PIM. Having opportunities to gather, review, and incorporate 360 evaluations is an important step in self-awareness and leadership development. We present for the first-time evaluation data on 50 participants from the PQ-I and found significant differences among subdomains based on role and generational groups. Further studies are needed to examine the potential impact of these evaluations on patient care.

Figure 1 Professionalism intelligence model.

Figure 2 Forest plot of average differences in self-scores between physicians and non-physicians. Positive β means that physicians rated themselves higher, on average, than non-physicians. Negative β indicates that physicians rated themselves lower, on average, than non-physicians.