Effectiveness of an operating room master of science program based on Kirkpatrick’s model

Background: Evaluation is one of the most critical issues in the training process. Kirkpatrick’s model is one method in educational evaluation that can accurately evaluate programs. This study aimed to evaluate the effectiveness of an operating room master of scienc (MS) program om based on Kirkpatrick’s model. Methods: This descriptive, cross-sectional study was conducted using convenience sampling with 46 students and graduates from the operating room MS degree at the Shiraz University of Medical Sciences in the academic year 2020-2021. A valid and reliable questionnaire with 47 items in three levels of reaction, learning, and performance was used to evaluate the opinions of graduates and students. The survey was sent to their email addresses and returned to the researcher after completion. Results: Kirkpatrick’s evaluation model showed that the overall evaluation of the operating room program was 70.97 ± 9.32, which can be considered a moderate level. Students’ satisfaction with the program was moderate (68.00 ± 14.97); their learning (84.54 ± 15.69) and performance (84.30 ± 9.28) were evaluated to be at a reasonable level. Conclusion: Considering the average effectiveness of the whole program, it is suggested that the educational managers of the operating room department plan and attempt to increase their indicators by considering the weaknesses and strengths of the program.


Introduction
Higher education is one of essential components of the educational system and a critical element of human development in any country. 1 The quality of training programs is one of the chief concerns of the university system, and promoting quality requires evaluation. 2 In the medical education system, the evaluation of programs is of particular importance due to the need for training an experienced workforce to provide health services and treatment with the desired quality 3 ; the curriculum is considered valuable when valid evidence and information confirm its effects at both individual and societal levels. 4 The field of the operating room is a branch of medical sciences in which graduates direct the structure, equipment, and operating room processes; participate in optimal surgical performance and care before, during, and after surgery; and share their learning in research, service, and educational fields with the community. 5 The operating room MS program is newly established. Due to the interactive, dynamic, and challenging nature of training in the operating room, such as teamwork and interdisciplinary collaboration, instantaneous emergence of risk in the operating room, exposure to acute and busy situations, close and constant interactions, and simultaneous tasks of team members in surgery, there is a need to evaluate this novel program. 6 Evaluation is an essential source to obtain reliable information and assess it correctly; it provides beneficial information about how to plan and implement programs and is a crucial tool in evaluating the performance of educational centers. 7 If program evaluation is carried out correctly, expectations for both decision-makers and endusers will be clearer 7 ; both the current and the desired situation are depicted; a balance is provided between output and input, and opportunities and threats are identified. In addition, the possibility of maximizing returns on costs

Original Research
and preparing a program tailored to the real needs of the university and society is provided; as a result, evaluation aids in bringing satisfaction and program improvement to decision-makers and stakeholders. 8 Several models are used to evaluate the effectiveness of programs, the most important of which is Kirkpatrick's model. 9 The most popular evaluation models in recent years are based on a four-level educational evaluation model first proposed by Kirkpatrick. This model is a comprehensive, simple, and practical method for evaluating myriads of educational situations. Kirkpatrick defines evaluation as determining the effectiveness of a program and divides the evaluation process into four levels: reaction, learning, performance, and results. The reaction level measures how participants feel about the program and tries to answer questions concerning attitudes toward the program.
Evaluation at the second level goes beyond learners' satisfaction and is a measure of increased learning; measuring this level is more complicated than the first level. At the third level, the transfer of learning in the learners' behavior and performance is measured due to the education they receive. At the fourth level, an attempt is made to evaluate education results in social and natural environments. 10 Since curriculum planning is done from the beginning to the end in the context of evaluation, evaluation of programs, especially newly established programs, is highly important. Thus, the current study was conducted to evaluate the effectiveness of the operating room MS program based on Kirkpatrick's model. No comprehensive evaluation of this program has been carried out so far.

Field of the operating room in Iran
The operating room associate's degree was established academically in Iran in 1986.. Since 2008 the operating room program has been established at both undergraduate and graduate levels based on the country's needs and is being implemented in several universities. In 2010, members of the Planning and Evaluation Committee of the operating room began compiling an MS degree curriculum in this field.
The program was approved by the Supreme Planning Council of Medical Sciences in 2014 and has been operational since 2016. 5 The operating room MS program is a postbaccalaureate program, wherein holders of bachelor's degrees in the fields of the operating room, nursing, and anesthesia can participate in an entrance exam held by the Ministry of Health to gain admittance. The program consists of four semesters, during which students are required to pass 28 special credits in the first three semesters and four thesis credits in the fourth semester.

Material and Methods
This descriptive cross-sectional study was performed using convenience sampling with students and graduates of the operating room MS program at the Shiraz University of Medical Sciences in the academic year 2021-2022. The study population consisted of all operating room MS students and graduates in this field. The inclusion criteria for students were studying in the third semester of the program and above. The study's exclusion criteria were the reluctance to participate in the study or incomplete response to the questionnaire.

Data collection
Upon receiving the contact information of the graduates and students from the faculty education department and obtaining informed consent, the questionnaires were sent to participants' email addresses; these were returned to the researcher after completion.

Instrument
Data collection tools in this study included a demographic information questionnaire (age, gender, previous degree, work experience in the operating room) and a researchercreated questionnaire according to Kirkpatrick's evaluation model consisting of 47 items at three levels of reaction, learning, and performance to evaluate the effectiveness of the operating room MS curriculum. Responses were scored using a five-point Likert scale, from 1 (very low) to 5 (excellent).
Reaction level items measured participants' satisfaction with the curriculum content, instructors, and program (14 items). The level of learning was prepared following the approved program and evaluated through student self-assessment before and after the program (18 items). The level of performance appropriate to the professional duties of graduates was prepared based on the approved curriculum of the MS degree. Accordingly, the graduates of this field should be able to perform their roles in service, education, research, and management (15 items).
The validity of this questionnaire was measured using face and content validity. In face validity, both qualitative and quantitative face validity were used. The item difficulty, appropriateness, and ambiguity were measured to evaluate the qualitative face validity; 12 MS operating room students and seven professors in this field were consulted. According to the results, items 4, 6, 7, 13, and 14 were corrected at the reaction level.
The quantitative face validity of the items was calculated by measuring the impact factor with a 5-point Likert scale and using the opinions of 2 senior operating room students and seven expert professors in this group. After extracting the impact factor of the items for each item and considering the minimum coefficient as more than 1.5, items 8 and 11 at the reaction level and items 2, 3 and 4 at the learning level were eliminated. It should be noted that, according to of professors' and students' opinions at this stage, three items were added at the learning level. At the end of the face validity measurement, 47 items remained at three levels: the level of reaction (14 items), learning (18 items), and performance (15 items).
Content validity was measured by seven professors and four MS students in the operating room using the content validity ratio (CVR) and content validity index (CVI) of the items.
According to Lawshe, experts were asked to comment on each item at three levels: necessary, not necessary but useful, and not necessary for CVR measurement. 11 The minimum value of content validity ratio, according to this table, was 59% for 11 people. As a result, all items at three levels (reaction, learning, and performance) were higher than the minimum.
Seven professors , four MS students in the operating room and a 4-point Likert scale were used to evaluate the CVI . In this method, a content validity index score higher than 79% is considered appropriate, and all items were higher than 86%. At the end, 47 items remained at three levels: the level of reaction (14 items), learning (18 items), and performance (15 items). Cronbach's alpha was used to evaluate the reliability of the questionnaire. A Cronbach's alpha value of 0.87 was obtained after conducting a pilot study with 40 MS operating room students; the values above 0.7 are considered acceptable for social sciences research.

Interpretation of results
The sum of the scores at each level was calculated. It should be noted that to calculate the level 2 score, the sum of the pre-test scores is minus the post-test score to obtain the score of this section. An overall score was also calculated by summing all three levels. The score was calculated as a percentage, and the effectiveness of the score was considered to be 80-100% as good, 60-79% as moderate, and 40-59% as poor to equalize the scores at all levels of evaluation .
The data were entered into SPSS version 21, and descriptive statistical tests (mean, standard deviation, frequency) and analytical tests (paired sample t test) were applied to interpret the information.

Results
In all, 46 people participated in the study (response rate of 85%). The mean age of the participants was 27.35 ± 3.58, with a range of 23 to 37 years. Other demographic characteristics are given in Table 1.
The overall evaluation of the program was found to be at an average level (70.97 ± 9.32). The evaluation of the program at the reaction level, which assessed participants' satisfaction, was found to be at a moderate level (68.00 ± 14.97). At the level of learning, required knowledge and skills were found to be at a good level (84.54 ± 15.69). Similarly, the level of performance, which indicates the performance of the individual in service, educational, research, and management roles, was also found to be at a good level (84.30 ± 9.28).
At the learning level, the results of a paired t test showed a significant difference in the evaluation scores from the pre-test (54.18 ± 13.13) to the post-test (84.40 ± 15.84) after passing the program (P < 0.001); therefore, students felt that participating in this program increased their knowledge and learning.

Discussion
This study aimed to evaluate the effectiveness of the operating room MS program based on Kirkpatrick's model. Based on our findings, the program evaluation of the operating room was considered moderate. According to the results of the first level of Kirkpatrick's model in this study, participants' satisfaction with the implementation of this program (instructor, curriculum, and the course) was reported to be at a moderate level; however, this program was evaluated as good in increasing participants' learning and performance. This finding indicates the necessity of paying attention to the selection of course instructors and up-to-date educational content; using new educational technologies; and establishing order and coordination in implementing the training course. As in the study conducted among dental students, the need for educational interventions in the dental curriculum has been emphasized. 12 Since the MS program in the operating room has been established for the first time in the country, it is natural that there are problems in terms of quantity and quality in this field. However, some of these problems will be solved over time with an increase in the number of graduates, especially in the postgraduate program in this field. On the other hand, the overall situation of the program would be improved by increasing the indicators that promote students' satisfaction and learning.
Ahanchian et al found that the level of satisfaction of more than half of nursing undergraduate students with an internship program was moderate. According to the students, lack of proper management of internships in the field, confusion of students during attendance in the department, duplication, the insignificance of the tasks given to students, and lack of a proper evaluation system were among the effective reasons for their dissatisfaction. 9 In another study, lack of attention to learning style and adoption of appropriate teaching methods were among the factors of participants' dissatisfaction. 13 However, Mohammadi et al reported that student satisfaction with a nursing doctoral program was at a reasonable level, 4 which can be related to the different nature of a specialized doctoral program compared to a master's program and the differences between learners in these two programs. The evaluation of the second level of Kirkpatrick's model also showed that the participants reported their level of learning increased after the program, and was reported to be at a reasonable level. In line with the current study, the results of other studies have shown that the changes in learners' level of knowledge and skills were effective after passing the program. 4,[13][14][15][16] Contrary to the present study, and according to Ahanchian et al, the internship program for undergraduate nursing students did not have the necessary efficiency in improving their technical and communication skills. 9 However, given that undergraduate nursing students in this program may not have the opportunity to perform or repeat many clinical skills, may not accurately estimate their skills.
Lee and Song, based on Kirkpatrick's model, found that despite high academic achievement, high employment rate, and high satisfaction with the program, undergraduate and postgraduate nursing students reported a lack of preparation for clinical situations. 17 In the present study, the study's findings of the changes in the behavior and performance of the participants of this program (Level 3 of the Kirkpatrick model) also showed that the program's learners were at a reasonable level. In other words, the participants felt they had achieved both the general and specialized competencies of the program. In line with the present study, Mohammadi et al demonstrated that Ph.D. students in nursing were at a reasonable level in terms of performance. 4 Bijani et al also found that more than half of nurses scored well after a continuous training program to prevent needle stick occupational exposure. 15 Heydari et al observed a significant difference in participants' performance before and after a workshop on the new teaching and learning methods. 16 Shayan and Nowroozi Rad, which also evaluated the effectiveness of the program on staff behavior using 360-degree evaluation from different perspectives (supervisors, peer colleagues, less experienced colleagues, patients, and the participant in the study), found that the program was effective at the performance level so that the knowledge and skills learned in the workplace were used from the perspective of all groups. 13 One of the strengths of this study was the use of Kirkpatrick's model as well as the views of students and graduates. Failure to implement the fourth stage of Kirkpatrick's model was one of the limitations of the present study, as it was not possible to examine the program's results in the community and natural environments in this time frame. On the other hand, there was no similar study to evaluate this training program to compare the results. In addition, the researchers compared the results of the present study in terms of satisfaction, learning, and performance only with other educational studies that used Kirkpatrick's model. Therefore, it is suggested that the researchers use Kirkpatrick model to determine the effectiveness of programs in similar studies.

Conclusion
The results of the study showed that the MS operating room program was moderately effective among learners; however, it is suggested that the operating room department's training supervisors plan to achieve higher performance indicators in this program, especially in student satisfaction, since learners' satisfaction may increase their motivation to learn. The findings of this study can help professors and heads of departments in the faculties and universities gain a realistic insight into the operating room master's program and pave the way for necessary changes to increase the program's productivity. Moreover, heads of departments and managers of hospitals and operating room staff can be aware of the results of this study in order to gain more knowledge about the effectiveness of the program in the operating room in the field of management and treatment and take steps to facilitate staff training at the master's level.