Bridging the Gap Between Competencies and Uncertainties in Postgraduate Training in Family Medicine: Results and Psychometric Properties of a Self-Assessment Questionnaire

Introduction

With the enactment of the National Competence Based Catalogue of Learning Objectives for Undergraduate Medical Education (NKLM) 2.0 on April 27, 2021, a nationwide consistent basis for a medical education curriculum was introduced in Germany.^1,2 It defines a clear graduate profile that should be reached by newly qualified physicians.²

Skills in managing personal and emotional challenges of work, uncertainty and change become more important than ever: In the midst of the COVID-19 pandemic, healthcare and educational climate has been increasingly experienced as volatile, uncertain, complex and ambiguous.^3,4 The uncertainty of a safe workplace with possible infection and the perception of inadequate protection measures is associated with the experience of psychological distress by health professionals.^5,6

Moreover, there are situations that are accompanied by uncertainty, since disease origin or accident mechanism is not traceable.^7,8 Similarly, the exploration of social-emotional aspects is difficult.⁹

Uncertainty is also present in patients or might be passed on to patients who – without necessary reassurance – experience higher levels of stress with a significant symptom burden.^10,11

Neuropsychologically speaking, uncertainty results in a stress response aiming at finding a strategy to safeguard one’s future well-being.^12,13 When acute stress responses cannot solve the state of uncertainty, the situation may become chronic persistently activitating neuroendocrine, cardiovascular, neuroenergetic, and emotional responses.¹² This ultimately leads to turbulences in the blood flow of coronary and cerebral arteries, hypertension, atherogenesis, cognitive dysfunction and depressed mood.¹² Uncertainty stress is associated with all categories of unintentional injury being more dangerous than life stress.¹⁴ Intolerance of uncertainty is associated with a higher perception of stress.^15,16

Dealing with uncertainty has been described as “one of the essential elements of primary care practice”.^16,17 High experiences of distress and low quality of work life results in rural general practitioners considering to leave rural practice.¹⁸ This is concerning with regards to the growing need of general practitioners, especially in rural areas.¹⁹ Weaknesses in postgraduate training in general practice were identified as one reason for this shortage.²⁰ Consequently, in order to ensure good patient care in the future, the improvement of the 5-year lasting postgraduate training in general practice in Germany receives increasing political attention.²⁰ Lately, a study has shown that one of the most important reasons for postgraduate medical trainees not to choose working in rural areas is clinical insecurity bonded to lack of professional competence.²¹ Therefore, in order to meet the growing need of physicians, the examination of the following two aspects in postgraduate training deserve attention: uncertainty and teaching of competencies.

A proper preparation for uncertainty might prevent from “disastrous consequences for practitioners”.²² By teaching skills and knowledge, uncertainty deriving from lack of ability can be tackled. Over the last decades teaching skills and knowledge has evolved into teaching competencies. Competence has been defined as...

the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and community being served.²³

Therefore, the implementation of competence-based learning does not exclude teaching of skills and knowledge. Competencies rather complement the training of skills and knowledge. It is believed that competence-based training ensures better preparation for practice.²⁴ Because of that, nowadays, competence-based programs are wide spread and implemented while restructuring teaching curriculums.^25,26

The Canadian Medical Education Directions for Specialists (CanMEDs), implemented in 1996, currently constitute the most widespread framework for competence-based training in the world.²⁷ Its core elements are the CanMED roles – six key roles that comprise essential competencies in a physician’s professional practice.²⁷ These roles define the physician as communicator, collaborator, manager, health advocate, scholar and professional.²⁷

Since competence-based training complements the teaching of skills and knowledge, and the lack of skills and knowledge results in uncertainty, competencies can be utilized to access the topic of uncertainty. The aim of this study was to approach uncertainty in postgraduate training in general practice by developing a questionnaire that comprises competencies linked to uncertainty.

Materials and Methods

We performed a cross-sectional study with trainees in general practice in the German federal states Schleswig-Holstein and Baden-Württemberg.

Questionnaire

In order to approach the topic of uncertainty in general practice training, we developed a competency-based questionnaire.

The questionnaire included three different sections, which consisted of items addressing competencies based on pre-existing instruments, all assessing relevant competencies in postgraduate training: the German Professionalism Scale (Pro-D),²⁸ the “Heidelberger Niederlassungsbarometer”²⁹ as well as a questionnaire surveying the procedural performance of primary care physicians in Germany.³⁰

Section 1: Professionalism

Professionalism is one of the CanMED key competencies and comprises lots of sub-competencies, which could induce uncertainty. Sub-competencies potentially associated with uncertainty are amongst others the disclosure of physicians’ limitations that affect care, the social accountability and commitment to patient safety.³¹ In order to incorporate professionalism into the questionnaire we used the German version of the Nijmegen Professionalism Scale. This scale is originally a validated instrument with 93 items assessing family medicine trainees’ professional behaviour on four domains – professional behaviour towards the patient, towards other professionals, towards society, towards oneself.^28,31 The German adaption of the Nijmegen Professionalism Scale is the Pro-D with 67 items.²⁸ In order to identify items of the Pro-D that are associated with uncertainty, four think-aloud telephone interviews were held with postgraduate trainees. Three interviewees were male, one was female. All of them were undergoing their postgraduate training in the federal state of Baden-Württemberg, Germany, and participated in an accompanying training program called “Verbundweiterbildung plus” that offers amongst others educational seminars and structured rotation planning.²⁰ The interviews were recorded digitally and transcribed verbatim. The items were followed through by the interviewer (MM) asking the following questions in the respective order: 1) Does the item have something to do with uncertainty? 2) If yes, why? Those items on which all interviewees agreed on having something to do with uncertainty were used for further evaluation. Due to ambiguity 2 out of those 12 remaining items were not used. Following one interviewee’s suggestion one item was split into two, in order to gain specification. In total, 11 items were incorporated into the section “professionalism” of the final questionnaire. Items were answered on a 4-point Likert scale from “frequently” to “seldom” regarding how often the professional behaviour was shown by the participant. “No information” was offered as fifth reply option.

Section 2: Personal Occupational Attitude

The section “Personal Occupational Attitude” contains five items representing a subset of an instrument, which is called “Heidelberger Niederlassungsbarometer”. This instrument was developed following the CanMED roles and assesses the acquisition of competencies during postgraduate training.²⁹ Additionally, it contains questions regarding the willingness to set up a practice.²⁹ Five general items concerning the acquisition of competencies were chosen in order to approach the topic from a broad perspective. The level of agreement with the items’ content was assessed on a 4-point Likert scale from “absolutely correct” to “not correct at all” next to the option of giving “no information”.

Section 3: Procedures

The five medical procedures chosen for the section “procedures” are a subset of 55 items that are performed by more than 50% of the general practitioners in Germany.³⁰ However, those five procedures can only be handled by less than 40% of the postgraduates in their last year of training.³² The level of security in performing these procedures was assessed on a 4-point Likert scale from “very secure” to “very insecure” supplemented with the reply option “no information”.

Sociodemographic Data and Free Text Comment

Sociodemographic data included sex, year of birth and year of postgraduate training. A free text comment was introduced with the question “What would help you to reduce uncertainty in your medical routine?”

The final questionnaire is available as translated English version from Additional file 1.

Recruitment

The final questionnaire was distributed to all participants of postgraduate training seminars in Baden-Württemberg, Germany in October 2016 as well as in December 2016 in Schleswig-Holstein, Germany. Participants were postgraduate trainees in general practice that attended accompanying educational seminars.

Statistical Analysis

The data analysis was carried out using SPSS version 26.0 (SPSS Inc., IBM). Descriptive statistics were conducted for item responses and sociodemographic data. Subgroup analyses regarding participants’ gender (male or female) as well as participants’ region of training (Schleswig-Holstein or Baden-Württemberg) were performed using Mann–Whitney-test since responses were not normal distributed. A partial rank correlation analysis was calculated for measuring the degree of association between year of postgraduate training and items’ responses. Participants’ age and gender were chosen as control variables. In case of missing or incomplete data, listwise deletion of subjects was applied.

The examination of psychometric properties included an item analysis with the descriptive statistics of mean, standard deviation, variance, skewness and kurtosis as well as the calculation of item difficulty. Preferable are item difficulties between 20% and 80%.³³ Corrected item-total correlations with values less than 0.3 specified items that did not correspond well with the overall scale.³⁴

Exploratory factor analysis tested the construct validity of the final instrument. A principal component analysis (eigenvalue>1, varimax rotation) with extraction of component loadings was carried out in order to describe the dimensionality of the domains. Sample and item suitability were evaluated with the Kaiser–Meyer–Olkin (KMO) criterion and the measure of sampling adequacy (MSA) for each item, respectively. Bartlett’s test examined sphericity.³⁴ Component loadings λ > 0.3 led to the assignment of the item to that particular component. Cronbach’s α described the reliability as a measure for the internal consistency.³⁵

The probability of error was set at a 5%-level (p<0.05).

Ethics Approval

This study was carried out in consideration of the Declaration of Helsinki. The realization of this study was approved by the ethics committee of Heidelberg University (Reference Number S-570/2015).

Results

The response rate was 85% (105/124). In total, 66 participants filled in the questionnaire in Baden-Württemberg and 39 in Schleswig-Holstein. Seventy-five per cent were female, 31% in their fourth and 32% were in their fifth year of training. The average age was 36 years. More sociodemographic data are presented in Table 1.

Table 1 Participants’ Sociodemographic Data (N=105)

Descriptive Statistics

More than 60% of the participating postgraduate trainees stated to show all professional competencies mostly or frequently. Respecting the patient’s self-determination was the item stated most frequently as “mostly” or “frequently” shown (N=104, 99%), followed by being able to learn out of mistakes (N=103, 98%) and openly approaching patients with a different frame of reference (eg, religion) (N=101, 96%). More than one-fifth of the participants answered that they seldom or sometimes showed the following skills: ability to balance work and life (N=25, 24%), letting a mild disorder (eg, tiredness) run its own way (N=24, 23%) and being able to conduct interventions that decreased aggression from the patient (N=22, 21%). Further details are shown in Figure 1.

Figure 1 Participants’ assessments of items in the section “Professionalism” (data labels show absolute frequencies). Note: Items reproduced from Roos M, Pfisterer D, Krug D, et al.Adaption, psychometric properties and feasibility of the Professionalism Scale Germany. Z Evid Fortbild Qual Gesundhwes. 2016;113:66–75. doi:10.1016/j.zefq.2016.04.002.28

In the section “Personal Occupational Attitudes” more than three-quarters of the participants stated all items as being correct or absolutely correct. Having gained an attitude that enables oneself to preserve lifelong learning was most frequently answered as being absolutely correct by 45% (N=47). The item least frequently answered as absolutely correct was dealing with diagnostic uncertainty (N=13, 12%). Further information is shown in Figure 2.

Figure 2 Participants’ assessments of items in the section “Personal Occupational Attitudes” (data labels show absolute frequencies). Note: Items reproduced from Kötter T, Rose SI, Götz K, Steinhäuser J. [The “Allgemeinarztbarometer A” – an instrument to assess primary care competencies during medical education and training]. Z Evid Fortbild Qual Gesundhwes; 2020; 155:34–39.64

The results in the section “Procedures” revealed that apart from the removal of cerumen obturans, more than half of the participants felt insecure or very insecure in performing the given procedures. Less than one-third of participating trainees felt very secure or secure in performing routine child check-ups (“U3-U9”) (N=7, 7%), youth check-up (“J1”) (N=30, 29%), application of a below elbow backslab (N=29, 28%) and partial removal of a toenail (N=24, 23%). The removal of cerumen obturans is the procedure that the participants answered most frequently to feel secure or very secure in performing (N=48, 46%). More than 10% of the participating physicians answered the five items concerning procedures with “no comment”. In Figure 3, the relative frequencies of answers in section 3 are displayed.

Figure 3 Participants’ assessments of items in the section “Procedures” (data labels show absolute frequencies).

Subgroup Analysis

Participating trainees in Schleswig-Holstein were significantly older and in more advanced training years than participants in Baden-Württemberg. However, subgroup analysis resulted in no statistically significant differences in response behaviour.

Partial Rank Correlation Analysis

The ability to do a constructive appraisal interview (rs=0.272, p=0.010), to let a mild disorder run its own way (rs=0.348, p=0.001), to psychologically process mistakes (rs=0.352, p=0.001), and to conduct interventions that decrease patient’s aggression (rs=0.292, p=0.006), were 4 out of 11 items in the first section of the questionnaire that were positively statistically significantly correlated with year of postgraduate training.

In the section “Personal Occupational Attitudes” a statistically significant positive correlation could be measured in all items except “gaining an attitude which enables lifelong learning” (rs=0.189, p=0.076).

A positively and statistically significant correlation in the section “Procedures” could only be seen in the performance of youth check-up (“J1”) (rs=0.356, p=0.001). The levels of significance as well as the correlations coefficients of all items are shown in Table 2.

Table 2 Item’s Correlation Coefficient and Significance Level Regarding the Correlation with the Year of Postgraduate Training

Determination of the Psychometric Properties

Descriptive item analyses resulted in means between 1.53 and 3.76. Lowest means were seen in the procedural performances, eg, routine child check-ups U3-U9 (1.53) and partial removal of a toenail (1.99). The variances ranged from 0.237 to 1.011 with highest variances in the procedural performances, eg, removal of a toenail (1.011) and application of a below elbow backslab (0.986). High skewness and kurtosis was seen in the item of respecting the patient’s self-determination (−2.462 and 8.478, respectively). Corrected item-total correlations ranged between −0.003 and 0.575 with the items 1.02 to 1.09, 2.04 and 3.03 under 0.3. Item difficulties were not favorable for the performance of routine child check-ups U3-U9 (17.66), enabling lifelong learning (81.66), learning out of one’s mistakes (87), respecting the patients’ self-determination (92), approaching patients with a different frame of reference openly (86), and the ability to obtain informed consent (82.67). More details are displayed in Table 3.

Table 3 Descriptive Statistics of Questionnaire Items

Taking the results of the item analyses into consideration we did not take the items 1.02–1.05, 1.08, 1.09 and 2.04 into consideration for exploratory factor analysis. The factor analysis with the 14 remaining items resulted in poor MSA for the items 3.01 (routine child check-ups U3-U9) and 3.02 (youth check up J1). Both items were deleted from factor analysis resulting in a final instrument with 12 items belonging to three factors with four items each. Those factors were specified as “doctor well-being and resilience”, “communication” and “skills”. Cronbach’s α of the final 12-item instrument was 0.736. Cronbach’s α of the three factors ranged between 0.528 and 0.691. The results of the exploratory factor analysis are presented in Table 4.

Table 4 Structure of the Final 3 Domain Instrument for Measuring Uncertainty in Postgraduate Family Medicine Training as Emerged from Analysis (12 Items, Cronbach’s α=0.736)

Discussion

Following the thesis, that lack of knowledge and skills results in medical uncertainty,³⁶ this study approached the topic of uncertainty in postgraduate training by surveying competencies regarding professionalism, personal occupational attitude and practical skills performed by postgraduate trainees. To our knowledge, this was the first study that related the concept of uncertainty with the training of competencies in postgraduate training. Competencies related to uncertainty were extracted from literature and interviews with postgraduate trainees in family medicine. The evaluation of the psychometric properties of the 21-item questionnaire resulted in the final 12-item instrument with acceptable psychometric properties.

Positive correlations between years of training and self-assessed competencies might indicate that the components “doctor well-being and resilience”, “communication” and “skills” can be acquired in postgraduate training.

Participating physicians were 36 years old and 75% female. These data show slightly more female trainees than in the participant description of the evaluation of postgraduate training in Germany (71% female).³⁷

The fact that the responses between participants in Schleswig-Holstein and Baden-Württemberg did not differ, although trainees were older and more advanced in Schleswig-Holstein, might be due to firstly, younger graduates tend to be more confident in skills and in knowledge than older trainees.³⁸ Secondly, differences in the training program might be beneficial for the participants in Baden-Württemberg. The “Verbundweiterbildung plus” is a curricular-based program for postgraduate training in general practice that started already in 2009 in Baden-Württemberg, whereas it just has started in 2016 in Schleswig-Holstein.²⁰ Essential components of the program are teaching seminars, the consideration of the German competence-based curriculum general practice and mentoring for trainees as well as train the trainer seminars for supervisors.²⁰ Participants in Baden-Württemberg might therefore have experienced an advantage in training.

The inability of dealing with diagnostic uncertainty in the descriptive results corresponds to the observation that postgraduate training fails in teaching trainees how to handle diagnostic uncertainty although it is inevitable in diagnostic activity.^39,40 For the sake of patients, research and the health care system, certainty is being proclaimed instead.³⁹ Though diagnostic uncertainty deserves to be mentioned as a statistical phenomenon of medical testing.⁴¹ The disclosure of uncertainty introduces sincerity into the patient–physician relationship.⁴⁰ The teaching of the fact that it is impossible to always be right, the teaching of humanity as well as the appreciation of the physician’s ability to disclose uncertainty should be introduced into every day’s postgraduate training.⁴⁰ Likewise, the coexistence of system failures and cognitive errors contributing to diagnostic errors have been described, and the informed patient as source for reducing diagnostic errors has been highlighted.⁴² Hence, training in communication seems even more important. The skill of doctor-patient communication might reduce the risk of decreasing patient satisfaction by sharing uncertainties in a calm, reassuring, empathic, untroubled appearing manner.^43,44 Physicians rarely communicate uncertainty to patients although it is described as critical element of shared-decision making since clinical decisions include uncertainty.⁴⁵

Regarding the high rate of trainees that answered to feel insecure in performing the chosen procedures, postgraduate training appears not to prepare trainees sufficiently for their everyday practice experience. Thus, it is an obligation to physicians as well as to all patients to consider changes in the teaching of procedures during postgraduate training. Procedure-based assessment (PBA) and direct observation of procedural skills (DOPS) as assessment tools can support the procedural training process.⁴⁶ In order to prevent skill decay, Kelc et al proposed cognitive training as “scientifically proven skill acquisition and retaining tool” for temporarily non-performing orthopedic surgeons.⁴⁷

Trainees’ competence in finding a balance between work and private life is not increasing during postgraduate training. Since this competence also depends on working conditions,⁴⁸ it must be noted that training in general practice in Germany is divided into two periods: at least 18 months training in a hospital setting and at least 18 months in the general practice environment.⁴⁹ Usually, physicians begin with training in the hospital. In the free text element, participants noted repeatedly that since they have started working in the practice, they would succeed more often in finding a work–life balance. In terms of keeping the workforce, fundamental changes concerning the working conditions in hospitals would have to be considered. Nevertheless, teaching of supporting skills in finding a work–life balance is essential in order to reduce uncertainty in both the hospital setting and ambulatory care.

Several answers to uncertainty have been described: coaching of students and practicing physicians, improvisational theater, teaching frameworks, entrustable professional activities (EPA), and the implementation of specific educational formats into the curriculum. In order to prepare students for uncertainty, clinical debriefs, interprofessional role plays, simulations, communication skills training, instructor emotional vulnerability, storytelling, and peer-to-peer conversations might be the most promising.⁵⁰

Coaching as an approach to reveal people’s potential resulting in the exhibition of their best performances has been described as possibility to overcome the impacts of uncertainty. A short and accessible coaching training approach of medical students and faculty members has been described as helping...

to mitigate against burnout, hold more rewarding, person-centred conversations in clinical practice and enable them personally to respond flexibly and constructively to change.³

Next to coaching for physician or student well-being and resilience, strong evidence was reported for coaching as an approach to improve technical skills.⁵¹

Furthermore, improvisational theater in health professions education was found to have positive impact on the comfort felt with uncertainty as part of the CanMEDS role Medical Expert.⁵² It might foster wellness and resilience by experiencing stress relief, self-esteem building, group bonding, sharing insecurities, adopting multiple perspectives, and responding free of judgement focusing on the awareness of what is happening.^53,54

Teaching frameworks like one-minute preceptor (OMP) and SNAPPS – acronym for summarize history and findings, narrow differentials, analyze differentials, probe preceptor about uncertainties, plan management, select case-related issues for self-study – facilitate the assessment of trainees, the integration of regular feedback and encourage self-directed learning.^55–57

In order to bridge the gap between that a trainee is able to perform the procedure confidently or that he knows about the corresponding competencies, one option might be the adoption of EPAs.²⁵ EPA-structured training is based on the concept that the supervisor decides which professional activity the trainee can be trusted with according to his or her level of competence.^58,59 Each so-called EPA represents “a real professional activity” and is underpinned by competencies anchored in competence-based frameworks.^25,60 Along postgraduate training EPAs are progressively achieved by the trainee – level by level – in order to master the full range of EPAs by completing specialty’s postgraduate training.^25,60 With these steps, uncertainties in trainees leading to dissatisfaction, medical errors or avoidance of activity execution might be reduced. Furthermore, EPAs could be useful in the early identification of trainees who are struggling in training.

In times of high economic pressure, trainees are often expected to work independently despite their degree of clinical uncertainty, which leads to their function in an “illegal-normal” state.⁶¹ This state can endanger patients’ safety and has an impact on the trainee’s well-being as well as learning process.^48,61 Regarding the need of physicians and the need for well-trained trainees, medical uncertainty should not be ignored but rather encouraged as “a critical prerequisite to a valuable medical practice”.²¹ By further improving postgraduate training regarding well-being and resilience, communication and skills trainees can gain competencies for reducing their uncertainty.

Strengths and Limitations

According to German Medical Association’s evaluation of 2017, 7282 postgraduate trainees were reported for financial support in the specialty training in general practice in 2016. Thus, our sample might have reached about 1.4% of all trainees in general practice.³⁷

Nevertheless, exploratory factor analysis was calculated with only a number of 64 data sets due to a high number of missings. Hence, further testing of the final 12-item instrument is planned in subsequent studies.

The data acquisition was carried out at educational seminars within structured teaching programs wherefore participants’ self-assessed abilities and competencies might not be representative for postgraduate physicians in general due to a selection bias.

Furthermore, data acquisition was conducted using self-assessment questionnaires. The Accreditation Council of Graduate Medical Education considers self-assessment skills as an important component of professionalism as well as a core competence.⁶² Studies revealed wide differences in evaluating self-assessment skills of physicians: ranging from little to none correlation between self- and external assessment to good congruency between physicians’ and assessor’s evaluation.^32,63 Since the participants were trainees, they might not have reached professional competence of congruent self-assessment. Therefore, the results of this study can only be considered indicators. Results should be verified by external objective observation.

This study was performed cross-sectionally. That is why conclusions in behalf of the question whether certain competencies are learnt during postgraduate training were drawn by comparing different persons in different stages of training among each other. Based on the assumption of a mild variety of trainees’ characteristics in different years of training, this approach is valid, but results need to be interpreted with caution.

Additionally, in this study, career changers or fields of completed rotations were not considered. Thus, results especially regarding practical skills with pediatric, surgical and orthopedic characteristics can be not conclusive.

This study only considers competencies concerning professionalism, personal occupational attitude as well as procedural performance. Indeed, the actual set of demanded competencies in general practice is far broader. However, in the view of the authors, these competencies include some of the most important aspects of competency-based training.

Conclusion

In order to reduce uncertainty with all its consequences, a focus on the teaching of competencies regarding the physicians’ well-being and resilience, communication and skills has to be continued in postgraduate training. Competence-based teaching structures such as EPAs could serve as enabler for a more extensive implementation of competency-based curricula into postgraduate training.