Utilizing Evaluation and Development Frameworks to Engineer a College-Wide Evaluation and Reform of an Undergraduate Dental Curriculum

Introduction

Background

A contemporary dental curriculum should reflect, and adopt to, innovation and changes in dental education. And such perspective is realized by leaders from various dental schools who consider continuous reflection, improvement, and periodic review cycle of the curriculum to be an essential measure to equip students with the necessary tools to thrive in the 21st century and respond to a fast-changing world.¹ Toward this end, others have published their experiences in conducting curriculum evaluation and implementing curriculum reform.^1–7 Those shared experiences present a valuable resource for reflection on how to bring about a curriculum reform implementation. However, they highlight contextual challenges associated with initiating curriculum change in the most valid and effective approach.

Although each individual institution reported a different trajectory of reform, similar themes can be deduced under the umbrella of systemic changes in dental educational models. The similarities emphasize comprehensive and patient-centered care provided through a reorganized curriculum that integrates basic and clinical sciences and adopts a competency- and community-based education as well as an instructional approach that fosters deep learning of students via curiosity, autonomy, independence, and responsibility.¹ On the other hand, curriculum evaluation with/without reform entails aspects of institutional uniqueness that reflects a broader complexity about its implementation. In fact, findings from education literature explicitly highlight the gap in curriculum reform implementation where there is a malalignment between intention and realization.⁸

The impact of published experiences for curriculum reform is essential to rationalize and realize. However, in the context of global dental education, there is scarcity of comprehensive analysis of curriculum reform implementation from a practical and universal perspective. Through the lens of multiple evaluation and implementation frameworks,^9–12 this paper aims to add to health professions education (HPE) literature by documenting how to initiate institutional change in an operational way and provide insights into the “change readiness” of dental colleges to proceed with curricular developments and innovations. Specifically, it underlines the importance of several key dimensions associated with planning and implementing a large-scale curriculum evaluation and how to translate these dimensions into tangible actions.

Context

An overview of the existing/evaluated curriculum will provide a clearer context to better understand the evaluation process and outcome. The existing curriculum consists of three phases over a six-year period, leading to a Bachelor of Dental Science degree (BDS), at King Saud University (KSU), Saudi Arabia. The first phase for all students is the premedical program year and permits the completion of prerequisite biomedical sciences for the dental curriculum and initiation of the general education requirements of the university. The second phase consists of the preclinical portion of the dental curriculum, which lasts for two years and includes the remaining general education courses. The curriculum is based on a traditional lecture-based format and preclinical laboratory experiences where students are exposed to the techniques of dentistry in traditional laboratory courses. The third phase consists of the clinical portion of the dental curriculum and lasts three years. The primary method of learning is by providing a required number of procedures to patients overseen by specialty faculty. Course assessments employ a variety of traditional formats including multiple-choice questions, essays, oral exams, and practical exams.

Aim

The evaluation process implemented at King Saud University-College of Dentistry (KSU-COD) was aimed to review the written, taught, and assessed college-wide curriculum to determine:

1. The effectiveness of each department in accomplishing its original goals and objectives.
2. The alignment with global, national, local standards and requirements.
3. Students’ performance in national and international licensing exams.
4. That the curriculum promotes active student engagement and meets diverse learning needs.
5. The incorporation of research-based best instructional practices and strategies.
6. That instructional strategies and assessment methods are aligned with the curriculum and current trends in dental education.
7. The satisfaction of stakeholders.

Materials and Methods

Before describing the methodology taken to evaluate the KSU-BDS program, it is essential to first chronicle the deliberate actions and processes taken by KSU-COD leaders which can be described via Kotter’s framework in transforming organizations.⁹ These actions are discussed more in-depth in the following text.

Pre-Evaluation Deliberative Processes

Problem Identification (Creating a Sense of Urgency)

The KSU-COD is well positioned as a leader in the region. The KSU-COD newly elected administration wanted to further enhance the institution’s position and reputation by attaining the Commission on Dental Accreditation (CODA). Therefore, the KSU-COD leaders decided to conduct a major curriculum evaluation including signing a formal contract with an external education consulting agency to guide and aid in the curriculum evaluation process. During their site visit, the external evaluators who have extensive knowledge and experience in program development and evaluation generated a detailed report. The gist of their report highlighted that the KSU-COD lacked an explicit plan in place for systematic ongoing review of the curriculum based on benchmarking CODA standards and outcomes assessment. The report also highlights the lack of basic teaching philosophy agreed upon collectively among the faculty and administration.

As a result, the administration, faculty, students, and alumni expressed concerns of such trajectory and stated desire to maintain the college as a preeminent dental program from an academic and scholarly perspective. The faculty and students have recognized the urgency to change, including the newly elected administration whose members were determined to deliver the COD updated strategic plan and vision (Box 1).

Box 1 Modified Strategic Plan, Vision, Mission, and Core Values for the College of Dentistry, King Saud University

Forming a Powerful Guiding Coalition and Creating a Vision with Strategies

To create buy-in, maintain institutional vitality and embrace diverse perspectives, leaders from the new administration created a Dental Education Division (DED) and hired five full-time junior faculty members to pursue postgraduate master’s and/or doctorate degrees in HPE. They also designated a Quality Control Committee (QCC) led by the Dean himself and included domestic external experts in HPE as well as diverse body of faculty, students and staff members representing each division. The committee has achieved the following:

1. Revised and modified on the institution’s mission, vision, core values strategic priorities, and promotional policies (Box 1) that reward teaching excellence for the institution to be consistent with the apparent change in the institution culture that is influenced by predominantly junior faculty members, well-rounded and vocal students, and supportive leadership.
2. Identified and created new set of competencies (Table 1) that are in parallel with the American Dental Education Association updated recommendations on best practice as well as the CODA requirements that have been considered, through consensus, to be the main goal for the new administration to meet the modified mission and vision of COD.

Table 1 Data Collection Methods (Environmental Scan)

Communicating the Vision and Empowering Others

To ensure alignment of support systems with the new vision, the new administration formed two committees to ensure effective communication, college-wide curriculum evaluation and attainment of change effort.

1. The Curriculum Oversight Committee (COC) to establish ongoing program evaluation procedures, report on opportunities for improvement of curriculum, and suggest maintenance, revisions, deletions, and/or creation of new curricula as needed.
2. The Faculty Development Programs Committee (FDPC) to develop necessary workshops for faculty, student, administrators, and staff members. The workshops were planned to be designed based on the identified needs and recommendations produced by the COC during and after completion of the curriculum evaluation process (ie, consolidating improvements and producing additional change).⁹

Evaluation of the Existing Curriculum

As mentioned earlier, the QCC signed a formal contract with a well-established consulting agency with experts in HPE from the United States, United Kingdom and Canada and commissioned the COC to conduct the evaluation. The COC was led by the director of DED and consisted of newly appointed faculty members at the DED, departments’ chairs, and administrator personnel, with periodic involvement of student representatives from each academic year. It also included two external evaluators from the consulting agency.

Procedure (Design and Data Collection)

A descriptive case study design was used with extensive multiple data collection methods that included literature review, document review of existing data, survey, focus group semi-structured interviews and observation of clinical and laboratory tasks (Table 1). Collection of qualitative data included 1) content analysis of several documents such as the evaluated curriculum, curricula of several international dental schools (Table 1), previous internal evaluation documents (open-ended responses within course and college-level satisfaction surveys etc.), and the outcomes of ad hoc reviews and previous national and international accreditations, 2) focus group interviews with multiple stakeholders (Appendix 1), and 3) field notes during real-time observation of clinical and laboratory work. Quantitative data included aggregate data of students and teachers’ feedback via Likert-scale questions of course evaluation surveys, American Dental Association’s (ADA) Survey of Dental Education and online survey questionnaires specific to KSU-BDS curriculum evaluation Appendices 2–5. The ADA survey, published in alternate years, reports the time devoted to the instruction, laboratory, and patient care activities from 59 United States dental schools in 3 major teaching areas: biomedical science; dental/clinical science; and behavioral, social, information, and research sciences. The overall data collection approach relied on multiple sources of evidence which provided a convergence of findings and allowed for a holistic view of the complexity of a college-wide curriculum evaluation.

While resource intensive, this descriptive in-depth design included different data collection methods, each with specific targeted participants and served specific purpose to detect the needs, interest, and responsibilities of stakeholders. Given the broad scope of evaluation purpose, it was challenging to concurrently address the various needs of stakeholders while maintaining a cost-effective evaluation design. Thus, it was important to implement a flexible, gradual, and needs-specific evaluation design. Such sequential implementation allowed the evaluators to address the purpose, questions and context while prioritizing a valid and practical evaluation approach. With a flexible evaluation design, it allowed for timely response to any emergent needs to be addressed as continuous refinement and reinforcement were expected and needed through the evaluation process.¹¹

Evaluation Model

The model developed for the purpose of the evaluation of the BDS curriculum (Figure 1) allowed for the collection of formative (process and outcome) as well as summative (outcome) data.¹³ Formative data was used to monitor the process of curricular change, to suggest and support additional changes to the curriculum, and to help understand what was done to achieve program outcomes by identifying gaps between curriculum outcomes and implementation objectives.^14–16 Furthermore, process evaluation data provided a context for interpreting the findings of the outcome and impact evaluation.¹⁷ On the other hand, formative outcome evaluation data primarily served to answer the question – To what extent were the outcomes objectives of the BDS curriculum were achieved?^18,19 All formative data were timely, concrete, and useful. Findings were communicated to department chairs, course directors, faculty, and students on a regular basis.

Figure 1 Undergraduate dental program evaluation model at King Saud University.

Summative evaluation data guided course directors to make decision about the overall merit (or worth) of their courses and to assess the achievement of outcome objectives.^11,15,20 When considering dissemination of the reporting, these data were used, for example, to decide upon the generalizability of curricular changes, the need for further restructuring of the curriculum, and/or the allocation of resources.^16,20 Since the attainment of accreditation was the driving force for the change initiative, summative data is essential, as external evaluators will use it for accreditation purposes.

Evaluation Approach

Three approaches surfaced through the evaluation process in a complementary manner. During the initiation phase, the Management-Oriented Approach (MOA) served decision makers (QCC and COC), which was crucial since the focus of the evaluation was to reflect the vision, mission, and core values of the new administration and to adhere to the updated BDS program competencies and CODA standards. In reference to attainment of CODA standards as another focus for the evaluation, expertise-oriented approach was important to use during specific stages of the evaluation. Also, the external evaluators helped in avoiding a common pitfall associated with MOA for which preference is usually given to top management.¹¹

Throughout the process, Participant-Oriented Approach (POA) was an overriding approach as COC members realized that utilization of evaluation results was significantly improved when there is greater collaboration with other stakeholders. During the evaluation, COC members consistently interacted with multiple stakeholders to obtain high-quality information, provide perspectives of important goals and objectives, and identify existing gaps in processes and procedures that were sometimes missed by evaluators.¹¹ POA was critical in increasing stakeholders’ sense of ownership and augmenting their understanding of evaluation, laying the foundation for an environment of ongoing program reflection and improvement. Although, time-consuming and labor intensive, this approach had the advantage of gaining support of key stakeholders to respond to recommendations for change.

Results

Based on the environmental scan described in Table 1, general and targeted needs of relevant stakeholders were identified (Box 2). The COC recommended that significant curricular changes are needed. The nature of such recommendation was sensitive regarding the success or lack thereof of some courses, evaluations of specific faculty and course directors, and recommendations to modify the existing curriculum by potential elimination and integration of courses. From a political perspective, the external evaluators help alleviate conflict of interest, provide constructive feedback when needed, and increase the objectivity in the presentation of the findings from the evaluation given their qualifications and extensive experience with transformational curricular evaluation and reform projects.

Box 2 General and Targeted Needs Assessment Outcomes

Discussion

This paper describes the anatomy and functions of a viable curriculum evaluation and reform implementation strategy characterized by the synchronized promotion of institutional accountability and individual agency with insights related to the internal dynamics of the top-down–bottom-up approach, proactive strategies conducive to institutional development, and the interplay between various global, cultural, political, societal, and contextual factors within a college reform.²¹ Upon reflection on the development and implementation of the KSU-BDS program evaluation strategy, it can be synthesized according to several key principles including collaborative, regulated, reflective, timely and student-centered, reliable and valid, continuous, and based on professional standards. These principles are described next in greater details.

The evaluation strategy of the BDS curriculum, which was executed by new administration, QCC and COC, took a collaborative approach. The evaluation had been, and will continue to be, a negotiated process.^15,22,23 It was characterized by a significant degree of collaboration among key stakeholders including administration, faculty, and students as well as consideration of patients and community, in both its development and implementation.²⁴ Because key stakeholders shared responsibility and decision-making, the evaluation was responsive to the needs of the BDS curriculum as well as those of stakeholders.¹⁵ By following a collaborative approach, notable promising initial results were noted as reflected by the increased stakeholders’ cooperation and involvement in the evaluation and receptivity to the findings. Such results are considered short-term wins,⁹ and such supportive culture shall be fostered to serve in building ongoing evaluation capacity within the COD.

As emphasized earlier, stakeholders’ involvement and engagement are essential to the success of the implementation process, especially in a complex environment such as the one described in this paper. The early steps of the envisioned curriculum, starting with the updated KSU-COD vision, mission, and strategy plan, were co-constructed with stakeholders, which was essential in building public support, aligning interests, and supporting individual and collective sense-making.²¹ Early and continuous faculty development workshops were designed and conducted for faculty members to provide them with the competences needed to understand, adapt, and shape the curriculum change.²⁵

The evaluation strategy involved a shift from a deregulated evaluation system previously administered by individual departments and courses to a more regulated system achieved through the creation of DED (ie, institutionalizing a new approach).⁹ It should be noted that the evaluation of the BDS program is the responsibility of the COC, which reports directly to the QCC. The regulation of the evaluation process facilitated the overall assessment of the undergraduate curriculum as well as planned curricular change in the future.^14,26 This approach highlights greater awareness toward aligning accountability mechanisms to the curriculum reform which is conducive to various stakeholders by allowing them to adjust their practices and focus on the implementation of the new curriculum. Furthermore, the evaluation was designed to promote reflective practice. As part of the reflective process, a new policy was proposed which requires course directors to respond to student feedback. Hence, future evaluations will be central to curricular change and ongoing curriculum development.^23,27

The importance of acknowledging and responding to feedback in a timely fashion was recognized by the evaluation approach which was predominantly POA. The evaluation strategy was characterized by considerable involvement from teachers and students. As such, it facilitated curricular improvement and student learning through the integration of the existing curriculum evaluation and future change processes.²³ Through an autonomy-centered evaluation process, the fundamental role of faculty members in curriculum development and implementation was recognized.²⁸ Specifically, faculty members’ feedback regarding curriculum and instruction is particularly relevant given their frequent interaction with students and understanding of students’ needs.²⁵

Students and teachers were, and will continue to be, actively involved in the ongoing evaluation and monitoring of courses and clinical rotations. They were encouraged to express their opinions and to provide feedback on content and pedagogical strategies as well as to make suggestions for improving the exchange of information. Currently, course directors, with the support of DED, are pilot testing rubrics and work-based assessment tools to be used for direct observation formative and summative assessment when students perform psychometric clinical and laboratory tasks.^29,30 It is also an example for short-term win of the assessment-based curricular changes.⁹ Such change was the result of an appropriate and prioritized response to students’ timely feedback. This will allow students to “witness changes to a course as they experience it, rather than moving on without ever knowing whether their recommendations had any affect”.²⁷

To ensure the reliability and validity of the findings of the evaluation, data and methodological triangulation were used.^31,32 Data were examined from different sources and over time and a combination of qualitative and quantitative methods were used. In addition, all evaluation instruments were designed in consultation with key stakeholders and external evaluators. Summary reports were presented to key stakeholders to validate the findings. The external evaluators were given high authority in the process, which ensured credibility by reducing biases as the QCC (led by the Dean) commissioned COC for review and reporting. According to Fitzpatrick et al, the greatest source of bias is when the evaluator is employed by the organization whose program is being evaluated.¹¹ The external evaluators accompanied two members of COC during the direct observations and solely conducted the focus group interviews. This allowed stakeholders to express their concerns freely. In addition to improving the evaluation outcomes, adopting an expertise-oriented approach provided political legitimacy and effective implementation of timely (early and continuous) stakeholder engagement.³³

Using meta-evaluation checklists recommended by the external evaluators, the evaluation was monitored on an ongoing basis by the COC and QCC to ensure that (1) the design was feasible; (2) activities were completed as planned and in a timely manner; and (3) instruments and products (data and reports) were of high quality.^11,16 The checklist comprised 49 items arranged under six overarching themes including conceptualization of the evaluation, sociopolitical factors, contractual/legal arrangements, technical design, management plan, and moral/ethical/utility questions.³⁴ The evaluation was modified as needed and as appropriate.

With regard to the revised curriculum, the following section provides an evidence-based and contextually appropriate perspective in attempt to guide future processes and decisions in implementing the revised curriculum. Guided by what have been described so far, Table 2 presents a comparison of the curriculum features between the existing/evaluated curriculum and proposed/revised curriculum. Through this comparison, and under the umbrella of guiding principles for the revised curriculum, an updated pedagogical paradigm is drafted that includes perspectives on content, instructional strategy, assessment methods as well as cultural and behavioral aspects. These foundational changes are important in moving the curriculum toward embracing the need for contemporary, empirically sound, and culturally sensitive dental care.³⁵

Table 2 Comparison of the Curriculum Features Between the Proposed Curriculum and the Current Curriculum

The goal of the updated pedagogical paradigm is to address deficiencies in the existing curriculum and prepare students to integrate knowledge and skills, develop professional attitude, and seek excellence in providing oral health care for society members. This pedagogical perspective is rooted in reflective critique of the evaluation process and outcome and through the adoption of established concepts in the context of HPE literature with regard to competency-based education, interdisciplinary collaboration, promotion of student-centered learning and implementation of valid, reliable, and feasible assessment methods.^36–39 To conclude, the development and implementation of the curriculum will be dependent on a whole-of-system approach with the provision of supportive leadership, adequate resources, effective communication, stakeholders’ consensus and commitment, and faculty development.

The DED will supervise and manage the evolution of the revised curriculum to warrant effective implementation as well as continuous improvement and alignment with the revised program objectives and updated competencies. With the help of the appropriate subcommittees (evaluation and assessment, IT/library resources, and faculty development subcommittees), the COC has created a centralized monitoring system evaluation to aid in monitoring the implementation of the revised curriculum and foster a robust transparent system for the curriculum reform. It depends on indicator frameworks (scores on internal examinations, results of national and international standardized examinations, performance-based assessments including an aggregate view on students’ e-portfolios, graduation questionnaire and exit interviews, course evaluations, and student advancement and graduation rates, etc.) tools (curriculum management system including curriculum mapping ie, the curriculum database) and processes (scholarly activities, ad hoc reviews, teacher training workshops, collaboration between dental and medical education departments at KSU, as well as with independent national and international agencies including the Saudi Commission for Health Specialties and ADA).

The intended KSU-BDS curriculum management approach is a methodical discipline of evaluating curriculum data (program and learning objectives, content, instructional approach including learning activities to session-, course-, and program-level objectives, and assessment) and making curriculum recommendations and/or changes. Utilizing export tools and gap analysis reports, the COC will continuously provide thorough yet targeted reports to the QCC and deanship office identifying any gaps or unforeseen redundancies as the curriculum delivery matures.

Implications and Recommendations

The deliberate actions and processes described in this paper highlight key implications in curriculum evaluation and reform implementation. The nature of this curriculum reform adopted a top-down-bottom-up approach as evident by the early involvement of relevant stakeholders.⁴⁰ This highlights the importance of having supportive leadership that plays a crucial role in driving and communicating the reform initiative, and more importantly, creates a culture that cultivates changes, and promotes professional learning communities through faculty development programs.⁴¹ Indeed, stakeholders’ early involvement, beliefs, individual and collective sense-making, and consensus, are pivotal to ensure successful implementation.^42,43

Although KSU-COD leaders engineered the evaluation process and subsequent reform recommendations, it was critical that relevant stakeholders co-constructed the process and perceived it to be autonomy-centered, advancing, and valuable, rather than prescribed, hindering or inconsequential.⁴⁴ The quality of sense-making influences the stakeholders’ degree of agency toward any curriculum change (ie whether it is embraced, challenged, or rejected).⁴⁵ However, galvanizing stakeholders’ support requires an effective policy design by developing a coherent implementation strategy that addresses the justification of the reform, clearness and consensus, and theory of change (ie the why, what, and how, respectively). As such, it implies developing a curriculum that intends to achieve clear objectives, according to a robust and attainable theory of change, and to realize a clear and valued educational vision. Furthermore, reform is costly and resource-intensive, and as highlighted in this paper, the implementation strategy should anticipate a high opportunity cost and needs to consider several influential factors such as available resources, existing capacity, and contextual factors.⁴⁶ Just as important, it needs to develop means to adjust those factors in the short term and plan for the long term to ensure its continuity. It is essential that the COD leaders maintains this continuity since CODA requires participating programs to compile and submit semi-annual resident milestones data, along with annual programmatic information such as faculty surveys and patient-log data. This emphasis on ongoing data collection and trend analysis further highlights the need for building organizational capacity for evaluation into the BDS curriculum by creating the DED. It also ties back into the importance of using the different complementary approaches in the evaluation plan. Finally, educators are encouraged to turn their contextual evaluation experiences into scholarly activities mainly via publishing focused research on discrete aspects of programs’ evaluation such as learners/teachers’ experiences, development and assessment of evaluation instruments, evaluation process, data collection procedures, and critical analysis of evaluation outcomes.

Conclusion

This paper unfolded the implementation of a curriculum evaluation and reform from an operational perspective, translating the abstract into concrete dimensions by contextualizing the principal factors in curriculum renewal and by drawing on real-time practices and examples from the evaluation process to show the utility of this work. The paper also recognized and addressed the unique institutional history, value, culture, and structure that impact the evaluation process and curriculum reform. Nonetheless, greater emphasis was placed on the general principles that remain applicable to other comparable contexts regardless of the distinctiveness in specificities.