Development and validation of evaluation tools of nursing students’ clinical pharmacology unit

Introduction

Owing to being practice based, as well as developments with regard to the information and caring techniques needed by nurses, education in nursing is being considered now more than ever.¹ Improving the quality of students’ performance in the area of education is one of the most important objectives for educators. More than half of a nursing training episode is dedicated to clinical education. During this period, students learn and improve upon clinical training in real situations, to prepare themselves for work in a clinical environment.²

Clinical care can be regarded as being of a high quality when the assessment of nursing skills is highly favorable.³ Clinical education has always been an important part of nursing education⁴ and faces many challenges.⁵ In general, it can be said that the training received by nursing students is far from the reality of their professional goals. Students are often not familiar with the expectations of others and the evaluation processes.⁶

This hinders familiarity with the execution of the nursing process, leading to lack of experience in undertaking correct clinical nursing procedures. It is also important to note that human life depends on nursing services.⁷ Nursing students often have difficulty engaging in theoretical discussions and providing quality care.

This impedes their ability to practice and apply the principles of basic training in clinical education, which means that they cannot integrate what they have theoretically learned in clinical environments. Such application is outlined in the evaluation criteria. Evaluation is one of the important elements of nursing education. By observing evaluation principles, the deficiencies and problems of nursing education programs can be determined.⁸ For this reason, a clinical skills evaluation form for students, which is based on their educational goals and measures, seems very important.⁹

Clinical evaluation could be likened to the compilation of multiple images of the moment of clinical practice for students. The greater the number of images produced, the greater the understanding of the results of the subject of evaluation. Challenges and frustration have always formed part of evaluation, demonstrated by the fact that 41% of nursing and midwifery students have complained about the evaluation process. A majority of students believe that clinical evaluation cannot identify students’ knowledge of theory and practice.¹⁰ In relation to this, instructors are seeking a reliable and valid tool for the clinical evaluation of students.¹¹

The clinical pharmacology unit was included in the nursing education curriculum in the 2014–2015 academic year. Along with the theory of nursing pharmacology in the second semester of nursing degree programs, this course is offered with the aim of creating a platform for the application of theoretical knowledge in the pharmacology field in real settings.

Owing to the lack of tools measuring students’ knowledge and application in practical units, a need was felt to develop a tool for this purpose. Therefore, this study aimed to develop a tool for the evaluation of the clinical pharmacology unit. Since students’ assessment is known to be one of the important aspects of clinical nursing education, the development and validation of a tool can give a better understanding of students’ strength in both theoretical and practical knowledge.

Methods

The present study used methodological research and was conducted in 2016 at Babol University of Medical Sciences.

Designing the questionnaire

At this stage, the development and validation of the tool were based on the stages proposed by Schwab.¹²

Development of items

This stage entailed the development of unique items. To develop the clinical pharmacology unit’s evaluation tool, a search was conducted on different databases, without a time limit, with the keywords “psychometric”, “tools”, “clinical education”, “pharmacology training”, and “nursing students”. The assessment tools found were in the area of the nursing management unit and other training units; no tools related to the clinical pharmacology unit were found. The review of articles indicated areas that needed to be addressed and special items that could help the present study. In addition to this, individual interviews were conducted with eight faculty members, to aid tool development.

Development of tools

At this stage, the researchers drafted the items. Then, a pool of items was selected (74 items were considered suitable for the item pool). Furthermore, items that could enable structural measuring were selected. At this stage, problems related to the design and structure of the tool were identified and corrected using a pre-test. During the pre-test of the items, the nature of the tool was tested, to ensure consistency with the goal.

The pre-test was conducted by having two experts in psychometrics and nursing education read the items a few times and review them. In the pre-test, the sample should ideally be representative of the population, comprising at least 300 subjects.¹³ In the pre-test, unclear items or those that are poorly worded were identified and corrected. Seventy items formed the primary pool of questions. Then, in the pretest, six items were removed because they were not consistent with the objectives of the study; 64 items remained.

Tool evaluation

At this stage, the tool was validated and the items analyzed. To validate the newly developed tool, face validity, content validity, and construct validity were measured.

Face validity

Face validity is determined using qualitative and quantitative methods. In terms of quality, face validity is concerned with the relevance of the tool, based on what it should seemingly determine. In terms of quantity, an impact score is used to determine face validity.¹⁴ In this study, to determine qualitative face validity, 10 nursing students were asked to comment about items’ level of difficulty and obscurity and the proportion of each item. Quantitative validity was determined by obtaining the impact score through surveys with the 10 students; the score indicated the importance of each of the questionnaire items. The subjects were asked to indicate the importance of each item on a 5-point Likert scale (from “very important” to “unimportant”).

Content validity

Content validity is the evaluation of the content of a test, considering the construct that it is supposed to measure.¹⁵ To determine content validity, qualitative and quantitative methods were used. The qualitative method entailed asking 10 experts in the field of tool making about the simplicity, clarity, and appropriateness of items, based on their location on the questionnaire. To evaluate content validity in a quantitative manner, a content validity ratio and an index were used. Then the experts were asked to give their opinions regarding the necessity and relevance of each item. To examine the validity of the two indices, the content validity ratio and the content validity index were used.

Content validity ratio

In this study, Lawshe’s model (1975) was used to determine content validity. At first, the tool was given to a panel of experts, to discuss its necessity. The experts’ answers were encoded as urgent, useful but non-essential, and non-essential. Then, the panel members’ votes were quantified through content validity. The values of the content validity equation ranged from -1 to +1.¹⁶ In this study, seven experienced professionals in the field of tool development were contacted by telephone or email. Then, the questionnaire was presented to them, so that they could assess the validity of the content. Items that were assigned scores of >0.42 by the experts of CVR assessment were retained as meaningful (p < 0.05).

Content validity index

The content validity index is the ratio of experts’ agreement about the relevance of each item, that is, the number of professionals who have assigned a score of 3 or 4 to each item divided by the total number of professionals. This essentially refers to the ratio of agreement on the relevance of each item.¹⁷ In this study, seven experienced professionals in the field of tool development were contacted by telephone or email. The tool was then sent to these professionals. According to Polit and Beck,¹⁸ a content validity index of 0.72 for five experts is good and that of 0.78 for six or more specialists is ideal.

Construct validity

Exploratory factor analysis was used to determine construct validity. In exploratory factor analysis, the researcher has no particular expectations about the number and nature of factors. Principal component analysis with orthogonal rotation is the most commonly used method to find factors. Eigenvalues that are greater than one standard deviation were used to extract the factors. In most studies, the appropriate factor loading has been set as 0.4. Nevertheless, items with a factor loading of at least 0.3 are maintained. Before factor extraction, to ensure that the tool’s items meet assumptions for the analysis of the main components, the Kaiser–Meyer–Olkin (KMO) test and Bartlett’s test of sphericity were conducted. The least proposed value of the KMO test was 0.6. Bartlett’s test of sphericity was especially used to determine if the correlations between test items equal zero.¹³

Reliability

Two methods, namely, internal consistency and test-retest reliability, were used to determine the questionnaire’s reliability. Internal consistency measures the extent to which individuals give responses that are stable over time. Cronbach’s alpha, with values ranging from 0 to 1, was used to assess internal consistency. For this purpose, 20 students were evaluated by their instructors during training, and three weeks later, the evaluation tool was completed on their behalf. Cronbach’s alpha was calculated for the overall utility and every aspect. In this study, 20 students were asked to complete the questionnaire on two occasions, to enable evaluation of consistency after 2 weeks and, therefore, completion of tool development. Then, intraclass correlation (ICC) was calculated for all domains and the whole questionnaire. The minimum acceptable value of ICC is 0.4.¹⁹

Study participants

In total, 264 second- and third-semester nursing students of the Islamic Azad University of Babol University of Medical Sciences participated in the study. The inclusion criteria included being a student and the willingness to participate in the study. For sampling, a census method was used in this study.

Moreover, the minimum sample size considered was five for each item. According to researchers, the sample size required for factor analysis, in order to determine construct validity, differs. The sample size recommended is five to ten samples for each tool.²⁰

To collect data in this study, a form containing personal information related to age, gender, and level of education was used.

Ethical considerations

This study was approved by the ethics committee of the University of Medical Sciences of Babol (approval number: 3388). All participants verbally gave informed consent to participate in the study. The confidentiality of individuals’ personal information was considered at all stages.

Results

The average age of the students was 20.23 ± 0.2 years. Further, 69.3% were female and 65.2% were in the second academic semester. Seventy-five items were assessed for validity. In examining the face validity, 11 options were removed, due to an impact score of <1.5. The number of items was subsequently reduced from 64 to 60, and the 60 items were included in the next step. To determine the content validity ratio, the views of experts in the field of tool development were considered. According to Lawshe’s table (1975), if the content validity value ranges from 0 to 0.62, then items with a numerical average of ≥0.5 are to be maintained. Two options were subsequently removed. Then, a 58-item tool was included in the next stage, which determined the content validity index. In this study, the minimum acceptable value of content validity was 0.72. Therefore, three items with a content validity index of <0.72 were removed, and 55 items were included in the heuristic content analysis stage. The KMO index in this study was 0.896. This indicated that there were sufficient data for analysis. Bartlett’s test of sphericity, with 12906.244, was also significant (p < 0.000). This indicates that the correlation between the items is sufficient and warrants factor analysis. Then, for determining the number of tool-producing factors, a scree plot and eigenvalues were used. The scree-plot graph showed that six factors were enough to explain the factors of the assessment tool for the clinical pharmacology unit. A 5-point Likert scale (“always” to “never”) was used for scoring. The scores ranged from 55 to 275.

Next, the tool’s factor structure was extracted through principal component analysis, with orthogonal rotation and varimax rotation (eigenvalue < 1). The result of principal component analysis was a factor matrix in which a factor loading for each item was determined separately. In this matrix, items that were highly correlated with each other were placed within one category or factor. In this study, the minimum factor loading for each item in the factor matrix and the rotated matrix was 0.3. In general, six factors were determined for the questionnaire, explaining 77% of the variance. After the extraction of factors, each was named based on the corresponding items and the extent to which these factors and concepts matched each other. Moreover, aspects of the evaluation of the clinical pharmacology unit that were determined in this study were evaluated (Table 1).

Table 1 Factors extracted from factor analysis using varimax rotation and factor loadings of their items Abbreviation: GERS, Global Expense Reporting Solutions.

First factor

This factor has eight items and refers to items related to Islamic norms, ethics, punctuality, and accountability. In this factor, the greatest factor loading was for the item “feels responsibility towards the educational needs of the patient and family” and the lowest factor loading was related to the item “is familiar with the laws and regulations of the ward and respects them”. The proportion of variance calculated for this factor before rotation was 41.694 and after rotation was 24.762.

Second factor

This factor has eight items and notes points related to effective communication. In this factor, the greatest factor loading was for the item “in patient education uses clear sentences”. The lowest factor loading was related to the item “adjusts to the environment and new situations and controls one’s emotions”. The proportion of variance calculated for this factor before rotation was 18.647 and after rotation was 23.033.

Third factor

This factor includes four items and refers to issues related to familiarity with medical terminology. In this factor, the greatest factor loading was for the item “reads the medication orders in the medical instruction correctly” and the lowest factor loading was related to the item “is able to read the label drugs such as name, size and amount of drug”. The proportion of variance calculated for this factor before rotation was 7.698 and after rotation was 7.233.

Fourth factor

This factor consists of 16 items and is concerned with nursing care prior to drug administration. In this factor, the greatest factor loading was for the item “knows high-risk drugs such as potassium chloride, digoxin, calcium, and insulin” and the lowest factor loading was related to the item “reads the drug and uses the expiration date from the medicine pack appropriately”. The proportion of variance calculated for this factor before rotation was 3.856 and after rotation was 4.950.

Fifth factor

This factor contains 11 items and is concerned with nursing actions during drug administration. In this factor, the greatest factor loading was for the item “is able to calculate the correct time of infusion through the micro set” and the lowest factor loading was related to the item “provides medication at the right time”. The proportion of variance calculated for this factor before rotation was 2.995 and after rotation was 3.736.

Sixth factor

This factor has eight items and is concerned with nursing care after medication. In this factor, the greatest factor loading was for the item “is able to detect drug interactions” and the lowest factor loading was related to the item “knows the right conditions to maintain the dissolved drug regarding environment temperature and the use date”. The proportion of variance calculated for this factor before rotation was 2.423 and after rotation was 3.554.

In this study, after determining construct validity, Cronbach’s alpha coefficient was calculated using a sample of 264 nursing students. A value of 0.96 was obtained, indicating that the questionnaire has good internal consistency. Then, Cronbach’s alpha was calculated for each factor. To determine the stability of the questionnaire in relation to reproducibility, ICC was calculated for all aspects. Given that the ICC of the tool is 0.91 and that of aspects ranges from 0.64 to 0.89, the reliability of the questionnaire is satisfactory (Tables 2 and 3).

Table 2 The results of the ICC coefficient (n = 20) Abbreviation: ICC, intraclass correlation.

Table 3 Results of internal consistency reliability (Cronbach’s alpha; n = 264)

Discussion

Evaluation is a process during which the skills and activities of students are evaluated. During this process, the strengths, weaknesses, and opportunities for improvement of skills and development are identified.²¹ The development of evaluation tools for the clinical pharmacology unit could highlight important issues. The present study presents the stages of the development and validation of the evaluation tool for the clinical pharmacology unit. The findings ultimately show that the instrument has satisfactory psychometric properties. Therefore, this tool can be used to evaluate the clinical pharmacology unit in the nursing education system. The tool’s items are scored using a 5-point Likert scale (“always” to “never”). The scores ranged from 55 to 275. In this study, in addition to the assessment of the quality of the content by panel members, content validity and the content validity index were calculated, to assess the tool’s content validity, which ultimately led to the removal of five items from the questionnaire. According to the results of exploratory factor analysis, the KMO index was found to be 0.896, and the higher it is, the better the factor analysis. In this study, given that the KMO value was higher than 0.80, it was considered good and the results were deemed favorable.²² Therefore, based on the results, the tool was classified into six domains or factors. Therefore, it seems that satisfactory results were as a result of careful selection of appropriate statements for the clinical pharmacology evaluation tool. With regard to the tool’s reliability, it should be noted that a reliable tool can increase the power of the study in the determination of associations and significant differences. In other words, reliability refers to an instrument’s stability. However, it should be noted that the reliability of a tool is closely related to its validity. In this study, in addition to appropriate calculation of the tool’s reliability separately, the final internal consistency of the tool was demonstrated by a Cronbach’s alpha coefficient of 0.96. Since an alpha coefficient of ³0.7 is considered appropriate,²³ the Cronbach’s alpha coefficient obtained showed the tool’s high internal consistency.

The tool also showed good reliability. The value of ICC coefficient was 0.91, which indicates good stability of the clinical pharmacology evaluation tool. In this regard, an ICC of ≥0.4 is considered satisfactory.²⁴

The final version of the extracted tool contained 55 items and six factors, namely, professional behavior, effective communication, knowledge of medical terminology, nursing procedures before administering medicine, nursing procedures during medication administration, and nursing care after administering medication. Six domains accounted for 77% of the cumulative variance, with minimum eigenvalues. In the domain of professional behavior, aspects such as respecting regulations, timely performance of tasks, ethics, and the rights of clients formed part of self-promotional activities. In the study by Pazargadi et al,²¹ the professional behavior domain had 14 items. In the evaluation tool of the University of Manitoba, which included two main domains, 12 sub-domains, and 55 phrases, this point constitutes one of the main domains known as professionalism and accountability, in turn consisting of three sub-domains and 22 phrases.²⁵

The effective communication aspect entailed items related to student’s interaction with patients, teachers, staff, and other students. In other studies, effective communication is one of the important factors considered in the evaluation of students. The provision of safe and quality care is dependent on nursing students’ ability to assess patients’ needs through the nursing process.²⁶

Nursing care before administering medicine entails items such as having enough knowledge and respecting essential points in the administration of medication to patients. Preparation by nurses is essential for the provision of effective and quality care to patients.²⁷

In nursing care during administration of medicine, emphasis is on points such as respecting six principles pertaining to the provision of drugs to patients (e.g., identifying drug interactions). Nurses’ attention at the time of administering drugs to patients is important for ensuring patient safety and yields positive outcomes of a treatment regimen.²⁸

Nursing care after drug administration entails issues such as creativity and innovation. During clinical evaluation, teachers should refer to activities such as critical thinking in clinical settings. In addition, they should ensure that students can apply critical thinking in clinical settings.²⁹

Knowing the side effects of drugs and their interactions are considered important points in the assessment of the clinical pharmacology unit. Lack of timely understanding of complications due to medication and ignoring laboratory changes associated with the diet treatment can threaten patient safety and disrupt quality care.

In general, it can be said that the clinical pharmacology unit evaluation tool is multi-dimensional and that the best tool for student evaluation is one that addresses various aspects. The development and validation of the clinical pharmacology assessment tool can be used to analyze the clinical operation of nursing students in educational systems.

Conclusion

In educational systems, the evaluation of students is an important measure of skill acquisition and provision of quality care to patients. In this study, along with the development of a valid tool to evaluate the clinical pharmacology unit, an attempt was made to assure the reader about the tool, by providing sufficient information about the process of evaluating the tool’s validity and reliability and the quality of its evaluation. For the first time in Iran, the development of this tool was based on the psychometric process and the views of nursing students and an impressive number and variety of specialists. The maintenance of simplicity and eloquence and consideration of brevity and the logical sequence of items are presumed to be the positive aspects of this tool. In this study, the tool has an acceptable factor structure and satisfactory reliability and validity. Therefore, it can be used in similar studies, in related topics, and with study populations in educational systems.

Acknowledgment

We are grateful to all faculty and nursing students who cooperated in this study.

Author contributions

Nasrin Navabi and Safieh Faghani have participated in data collection. Fatemeh Ghaffari has participated in data analysis and drafting the text. Abbas Shamsalinia has participated in data analysis. All authors contributed toward data analysis, drafting and critically revising the paper and agree to be accountable for all aspects of the work.

Disclosure

The authors report no conflicts of interest in this work.

References

1.		Latifi M, Shaban M, Nikbakht Nasrabadi A, Mehran A, Parsa Yekta Z. Effect of clinical evaluation with portfolio on critical thinking skills of nursing students. Iran J Med Educ. 2011;11(4):368–381.
2.		Assadi S, Shariati A, Haghighi S, Latifi SM, Sheini-Jaberi P. Effects of clinical education and evaluation with portfolio method on nursing students’ satisfaction: a clinical trial. J Clin Nurs Midwifery. 2014;3(3):70–79.
3.		Zhang L-F. From conceptions of effective teachers to styles of teaching: implications for higher education. Learn Individ Differ. 2009;19(1):113–118.
4.		Hickey MT. Baccalaureate nursing graduates’ perceptions of their clinical instructional experiences and preparation for practice. J Prof Nurs. 2010;26(1):35–41.
5.		Graceanne Adamo MAC, Hawkins R. Clinical evaluation in advanced practice nursing education: using standardized patients in health assessment. J Nurs Educ. 2002;41(5):215.
6.		Lützén K, Blom T, Ewalds-Kvist B, Winch S. Moral stress, moral climate and moral sensitivity among psychiatric professionals. Nurs Ethics. 2010;17(2):213–224.
7.		Pazargadi M, Ashktorab T, Khosravi S. Multi-rater evaluation in the clinical evaluation of nursing students: instructors’ experiences and viewpoints. J Qual Res Health Sci. 2012;1(2):102–111.
8.		Chang E, Daly J, Bell P, Brown T, Allan J, Hancock K. A continuing educational initiative to develop nurses’ mental health knowledge and skills in rural and remote areas. Nurse Educ Today. 2002;22(7):542–551.
9.		Lynette Cusack R, Morgan Smith R. Power inequalities in the assessment of nursing competency within the workplace: implications for nursing management. J Contin Educ Nurs. 2010;41(9):408.
10.		Hsu L-L, Hsieh S-I. Development and psychometric evaluation of the competency inventory for nursing students: a learning outcome perspective. Nurse Educ Today. 2013;33(5):492–497.
11.		Vaismoradi M, Salsali M, Ahmadi F. Nurses’ experiences of uncertainty in clinical practice: a descriptive study. J Adv Nurs. 2011;67(5):991–999.
12.		Schwab DP. Construct validity in organizational behavior. Res Organ Behav. 1980;2(1):3–43.
13.		Peterson C, Crosby R, Wonderlich S, et al. Psychometric properties of the eating disorder examination-questionnaire: factor structure and internal consistency. Int J Eat Disord. 2007;40:386–389.
14.		Lang TA, Hodge M, Olson V, Romano PS, Kravitz RL. Nurse–patient ratios: a systematic review on the effects of nurse staffing on patient, nurse employee, and hospital outcomes. J Nurs Adm. 2004;34(7–8):326–337.
15.		Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med. 2006;119:166.e7–166.e16.
16.		Lawshe CH. A quantitative approach to content validity. Pers Psychol. 1975;28(4):563–575.
17.		Rungtusanatham M. Let’s not overlook content validity. Decis Line. 1998;29(4):10–13.
18.		Polit DF, Beck CT. The content validity index: are you sure you know what’s being reported? Critique and recommendations. Res Nurs Health. 2006;29(5):489–497.
19.		Knapp TR, Brown JK. Ten measurement commandments that often should be broken. Res Nurs Health. 1995;18(5):465–469.
20.		Govender R, Schlebusch L. Suicidal ideation in seropositive patients seen at a South African HIV voluntary counseling and testing clinic. Afr J Psych. 2012;15:94–98.
21.		Pazargadi M, Ashktorab T, Alavimajd H, Khosravi S. Developing an assessment tool for nursing students’ general clinical performance. Iran J Med Educ. 2013;12(11):877–887.
22.		Munro BH. Statistical Methods for Health Care Research. Philadelphia, PA: Lippincott Williams & Wilkins; 2005.
23.		Gliem J, Gliem R. Calculating, interpreting, and reporting Cronbach’s alpha reliability coefficient for Likert-type scales 2003. In: Midwest Research-to-Practice Conference in Adult, Continuing, and Community Education. 2003; Indianapolis, IN.
24.		Terwee C, Bot S, Boer MD, et al. Quality criteria were proposed for measurement properties of health status questionnaires. J Clin Epidemiol. 2007;60(1):34–42.
25.		Leung SF, Arthur D. The alcohol use disorders identification test (AUDIT): validation of an instrument for enhancing nursing practice in Hong Kong. Int J Nurs Stud. 2000;37(1):57–64.
26.		Momeni H, Salehi A, Seraji A, Foroughi S, Hasheminia S. A study of patients’ satisfaction from clinical education of nursing students in selective general hospital in Markazi, Lorestan and Chahar Mahal-va-Bakhtiari Province in 1389. Stand J Edu Res Essay.2013;1(3):45–47.
27.		Neishabory M, Raeisdana N, Ghorbani R, Sadeghi T. Nurses’ and patients’ viewpoints regarding quality of nursing care in the teaching hospitals of Semnan University of Medical Sciences, 2009. Koomesh. 2010;12(2):e134–e143.
28.		Zadeh H. Managers’ and nurses’ role in ensuring safe patient’ care: nurses’ perspective. J Health Promot Manage. 2012;1(2):7–14.
29.		Duers LE, Brown N. An exploration of student nurses’ experiences of formative assessment. Nurse Educ Today. 2009;29(6):654.