Establishment of an Evaluation Index System of Competencies for College Senior Students in General Practice Medicine in Anhui Province, China

Background

With the rapid development of the economy and significant improvement in people’s living standards in China, it is urgent to develop general practice education.¹ To promote the development of general practitioner (GP) systems, GPs should form the core of general medicine talent.² GPs play an essential role in improving people’s health; thus, high levels of competencies are required for GPs.³ It is well known that colleges and universities are the primary gateways to train GPs. Over the past 20 years, general practice medicine and the role of GPs in medical education have aroused great attention.⁴ It is of great significance for the cultivation of general practice medical students’ ability and quality to promote the sustainable development of general medicine.

Currently, the establishment of a competence model for medical students has been increasingly optimized.^5,6 Valentini have found that it is feasible to apply competency-based general practice to postgraduate education courses Complementary medicine and integrative medicine competence play an important role in the training of general practitioners.⁷ Fielding’s team established the General Practice Registrar Competency Assessment Grid (GPR-CAG), which appears to have utility as a formative GP training WBA instrument.⁸ However, research on the comprehensive quality of college senior students in general practice medicine is rare, and evaluation methods suitable for measuring quality do not exist in China and cannot further guide the reform of talent cultivation and reform in general medicine in colleges and universities.

In this study, we co-opted the theories and methods of Bloom et al for the measurement of competencies of college senior students in general practice medicine.⁹ The competencies were divided into three aspects: knowledge level, working skill and professional accomplishment. Based on the survey results of the current situation and needs of GPs in Anhui Province, the evaluation index system of GPs’ competencies was constructed.

Materials and Methods

Subjects and Study Location

Anhui provincial health administrative departments, higher medical colleges, medical and health institutions at all levels and other relevant experts were selected for consultation. A total of 20 experts were chosen by random sampling methods, including 8 education researchers in general medicine of Anhui higher medical colleges, 3 municipal/county health administrative personnel, 5 principals of primary medical and health service institutions, and 4 primary GPs.

Study Methods

Initial Establishment of Evaluation Index System

The comprehensive competencies of GPs were based on the experiences of domestic and foreign scholars and combined with China’s national conditions and the actual local situation. The subtopics under each topic were identified by Bloom’s Taxonomy, and the levels of cognition necessary for learners’ ability were demonstrated. The ascending levels of memory, understanding, application, analysis, evaluation, and creation were defined. Finally, the evaluation index system of competencies for college senior students in general practice medicine was preliminarily constructed, including 3 primary elements, 10 secondary elements and 38 level-3 elements.

Delphi Method

The Delphi method was applied to perform two rounds of expert consultation.¹⁰ The first round of expert consultation consisted of the following: (1) collection of basic information of experts, including gender, age, education background, professional title, working years, professional occupation and familiarity with evaluation elements; (2) experts’ opinions on the initial evaluation index system,¹¹ which used a 5-point Likert scale to assess the importance degree of the index elements at all levels, which was divided into 5 grades, including very important, relatively important, general important, not too important and not important with 5, 4, 3, 2 and 1 points, respectively. In addition, there was a column for suggestion to supplement or modify the various elements. The second round of expert consultation consisted of the following: after the elements at all levels were revised and improved, the second expert consultation was conducted, and the importance of the elements was scored again. According to the analysis of two rounds of expert consultation, the index elements of the evaluation system were clarified.

Analytic Hierarchy Process (AHP)

AHP was proposed and established by the famous American operations research scientist Saaty. AHP is a common method used to evaluate the relative importance of influencing factors based on qualitative and quantitative analysis of their risks and benefits.¹² Evaluation of the abilities of college senior students in general medicine is a comprehensive process and involves many factors. Compared with traditional evaluation methods, this method establishes multidimensional evaluation standards with competencies for college senior students in general medicine as the core, and this method has a good degree of differentiation and considerable and reasonable evaluation data and plays a promoting and guiding role in the cultivation of competencies for college senior students in general practice medicine. Yaahp software is a visual modelling and calculation software based on the principle of AHP, providing functions, such as hierarchy model construction, judgement matrix and ranking weight.¹³

The main steps to construct the evaluation index system using AHP include the following:

Establishment of hierarchical model: The elements involved in the evaluation index system of competencies were divided into several levels, and a hierarchical model of multiple levels was developed.

Construction of the judgement matrix: The pairwise importance of elements was compared and analysed by comparing factors at the same level. The 1–9 AHP Saaty’s scale was adopted.¹⁴

Establishment of index weight: Yaahp software was used to calculate the ranking weight of the overall target and subtarget.

Consistency test: A consistency test was used to determine whether the weight allocation was reasonable. Yaahp software was used to perform the consistency test. When the consistency index CI of the judgement matrix was less than 0.1 and the random consistency ratio CR was less than 0.1, it was believed that the judgement of each weight of the index had no logical error, had satisfactory consistency, and passed the test.

Statistical Method

Excel 2010 software was used to process relevant data. SPSS 18.0 software and Yaahp 11.1 software were used for statistical analysis. Counting data were tested by chi-square test, P< 0.05 was considered significant.

Results

Basic Information on Experts

Twenty experts participated in this study, and their basic information is shown in Table 1.

Table 1 Basic Information of Experts (n=20, %)

The Active and Authority Coefficient of the Experts

The active coefficient is the active degree of experts who participated in this research, which is measured by the returns ratio of the questionnaires.¹⁵ In the first round of this survey, 20 questionnaires were sent out, and 20 were collected, yielding a recovery rate of 100%. In the second round, 20 questionnaires were issued, and 20 were collected, yielding a recovery rate of 100%. This finding indicates that the experts were highly motivated.

The authority coefficient of experts is mainly affected by the judgement basis (Ca) and the familiarity (Cs) of the elements.¹⁶ The judgement basis of experts was divided into practical experience (assignment 0.5, 0.4, 0.3), theoretical analysis (assignment 0.3, 0.2, 0.1), literature understanding (assignment 0.1, 0.1, 0.05), and subjective feeling (assignment 0.1, 0.1, 0.05). The three levels of expert familiarity were assigned 1.0, 0.5 and 0.0. The authority coefficient of experts is the arithmetic mean value of the familiarity coefficient and judgement coefficient, which is between 0 and 1. The higher the authority coefficient, the higher the authority degree of experts. In the first round, the coefficient of expert authority was 0.803 (Ca=0.830, Cs=0.775). In the second round, the coefficient of expert authority was 0.858 (Ca=0.845, Cs=0.870).

The Degree of Concentration and Coordination of Expert Opinions

The Kendall coordination coefficients (W) of the two rounds of expert consultation were 0.506 and 0.619, respectively, and the difference was statistically significant (P<0.01).

Results of the First Round of Expert Consultation

According to the opinions of the consulting experts, three first-level elements, 10 second-level elements and 38 third-level elements of the evaluation index system were modified as follows: (1) elements removed: “system analysis” in secondary elements was deleted. Five items in tertiary elements were also deleted: “using health information to guide the community working ability”, “analysis of the ability of health”, “the skills of using evidence-based medicine”, “the ability to apply the law to solve disputes” and “the ability to solicit advice and suggestion from others”; (2) modified elements: “basic pharmacological knowledge” was changed to “pharmacological knowledge and clinical rational drug use”, “health management knowledge” changed to “grassroots health management knowledge”, “good communication skills with the patient” changed to “good communication skills with patients and family members”, and “health records management skills” changed to “establishment, use and management of health records”. “The ability to carry out the work of teaching” and “the ability to launch scientific research work” were integrated into “scientific research”.

Results of the Second Round of Expert Consultation

On the basis of the first round of expert consultation, the evaluation index system was modified, and 3 first-level elements, 9 second-level elements and 32 third-level elements were formed. In the second round of expert consultation, the second-level element “continuous self-directed learning” was revised to “educational learning”. According to the opinions of two rounds of expert consultation and discussion by the research group, the evaluation index system was finally determined, including 3 first-level elements, 9 second-level elements and 32 third-level elements.

Evaluate Index Weight Results

Yaahp software was used to establish the hierarchical structure model, with the “evaluation index system of competencies for college senior students in general practice medicine” as the decision-making target, the first- and second-level elements as the middle layer, and 32 third-level elements as the scheme layer.

Yaahp software generates a judgement matrix of decision objectives and intermediate elements based on the model. Combined with the weight of experts, each judgement matrix was scored in group decision-making, and the weight results of each index element were obtained by calculation and analysis (Table 2).

Table 2 Weight of Evaluation Index System of General Practice Graduates’ Ability and Quality (%)

Discussion

College senior students in general practice medicine are future general practitioners, the competencies of whom can determine the future of basic health care in our country. Therefore, the establishment of an evaluation index system of competencies for college senior students in general practice medicine is of great significance. WONCA has issued a consensus statement on general practice/family medicine (GP/FM), which defines the discipline of GP/FM and the professional tasks of family physicians. In addition, WONCA describes the core competencies required by family doctors. Core competencies are essential for general practitioners, regardless of their applications in any healthcare system.^17,18 In this study, the Delphi expert consultation method and AHP method were combined to construct an evaluation index system of competencies for college senior students in general practice medicine. The two methods of application were based on the theoretical knowledge and practical experience of consulting experts. AHP was used to combine subjective evaluation with mathematical evaluation, and the data were statistically processed on the basis of subjective judgement of experts, all of which increased the logic and scientific nature of this study.¹⁹ The results of this study showed that CI, CR and the judgement matrix of index elements at all levels were all less than 0.1, suggesting that the ranking of each level exhibited satisfactory consistency, and the evaluation index and hierarchy structure constructed was objective and scientific.

After two rounds of expert consultation, 3 primary elements, 9 secondary elements and 32 tertiary elements were ultimately determined. The relative weight of each element was statistically analysed.

First, among the first-level elements, “professional quality” had the highest weight with a weight coefficient of 0.4261. Three secondary factors were established under this index, and the weight coefficient of “professional ethics” was at the top of the three with a weight coefficient of 0.2614. The results suggested that professional quality was the most important ability of college senior students in general practice medicine. The goal of general practice was to advocate the human-centred and biopsychosocial medical model as guidance, respect the needs of patients and families, pay attention to the physical and mental health of individuals and families, and pay attention to the care of the inner quality of life.²⁰ In the context of the poor doctor-patient relationship and increasing medical disputes, doctors’ professional ethics, as a soft service, were increasingly valued, which directly impacted the improvement of medical service quality.²¹ It is critical for college senior students in general practice medicine to strengthen their professional quality during professional learning. Under the influence of the role-oriented environment of doctors’ profession, college senior students gradually internalize and form stable and efficient professional ethics, which is a prerequisite for a good job as a general practitioner.

Second, weight coefficient of “community practice” was at the second highest level in the secondary factors with a weight coefficient of 0.1526. As the main force of primary medical care, GPs’ community practice skills are essential. With the reform of the medical system, a basic medical and health system covering both urban and rural residents should be established and improved, and the medical concept of “minor diseases are at the grassroots level, serious diseases are in hospitals, and rehabilitation is returned to the community” should be implemented.²² For this purpose, GPs need to be proficient in the use of general practice principles and integrate them in practice. According to the findings of this study, the weight coefficient of “High blood pressure, diabetes, mental illness and other health education skills” was the highest in “community practice” with a weight coefficient of 0.0475. The prevention and management of chronic disease is a priority for primary care services.²³ One of the most important skills of GPs, which was different from other physicians, was health education and the development of a high-quality preventive service, aiming to prevent diseases and improve residents’ self-management ability and quality of life.²⁴ Therefore, college senior students in general practice medicine should pay attention to the cultivation of health education skills when cultivating their competencies. In addition, among the secondary factors, experts agreed that communication skills are one of the important elements in evaluating competencies for college senior students in general practice medicine. In particular, the ability to communicate well with patients and their families has received increasing attention, and good and effective communication is the basis and guarantee for increasing patients’ subjective initiative. When potential conflicts occur between doctors and patients and between doctors and patients’ family members, doctors are also required to have a skilled communication and coordination ability to resolve conflicts. Therefore, good communication skills are an effective method to improve the service quality of GPs, and the learning of good communication skills is essential for college senior students in general practice medicine.²⁵

In addition, among the secondary factors, the weight coefficient of “educational learning” was the lowest with a weight coefficient of 0.0346. Four third-level elements were constructed in this factor. Among them, “scientific research” was the lowest to be required with a weight coefficient of 0.0049. This may be related to the scientific research system of general practice and the teaching and scientific research level of general practitioners in China.²⁴ The development of scientific research in general practice could improve the visualization of general medicine and the sense of achievement of GPs and facilitate the development of general practice medicine.²⁶ To improve the scientific research ability of GPs in different manners and establish a close relationship between clinical practice and scientific research, colleges and universities should strengthen the cultivation of scientific research ability for students in general practice medicine, improve their thinking ability for scientific research, enable them to combine their own characteristics when facing specific working environments and clinical problems and apply the scientific research results to practical work.²⁷

Conclusion

In this study, we constructed an evaluation index system of competencies for college senior students in general practice medicine. The evaluation index system of competencies for college senior students in general practice medicine was established with 3 primary factors, 9 secondary factors and 32 tertiary factors. The consensus on the content of the competencies of college senior students in general practice medicine suggests that these elements are necessary for those who will become general practitioners. This system can be used as the basis to evaluate the ability of college senior students in general practice medicine, and provide guidance for the cultivation and evaluation of general medicine talent. However, due to the limitations of the research scope and conditions, this research was only carried out in Anhui Province; thus, further in-depth measures are needed for nationwide promotion.