Designing Direct Observation of Procedural Skills ( DOPS ) Test for Selective Skills of Orthopedic Residents and Evaluating Its Effects from Their Points of View

© 2015 The Authors. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, as long as the original authors and source are cited. No permission is required from the authors or the publishers. Article History: Received: 16 Feb 2015 Accepted: 14 Dec 2015 ePublished: 1 Feb 2016


Introduction
Clinical education and assessment are very important in medical education and provide an exceptional opportunity for teaching required medical skills and promoting procedural skills in interactions with patients. 1Today in medical education, two important features are considered more than in the past: an emphasis on evaluation as a tool for ensuring the quality of education programs and moving toward outcome-based education in which the outcomes of learning are defined and content is developed according to the outcomes. 2,3][6] Regarding the necessity of assessment, it is required to use methods and tools of formative assessment for clinical skills in order to help students to learn and be prepared for final tests.One of the common methods for assessing procedural and instrumental skills is Direct Observation of Procedural Skills (DOPS), which involves the direct observation of students using clinical skills.In this method, the evaluator's observations are registered based on a checklist and real findings will be reflected to students.8] DOPS is an opportunity to provide students with constructive feedback, concentration and focus for using required skills, because evaluation, with the purpose of promoting performance, needs on-time and specific feedback.3 Beard et al. in 2005 9 and Wiles et al. in 2007 10 confirmed the results of DOPS. 11Also, based on the results of a study in Imperial College London, the reliability and validity of this method are appropriate in evaluating clinical processes. 12egarding the importance of applying appropriate and effective methods of assessment and the lack of studies in this field in Iran, the present study was designed and performed with the purpose of developing DOPS checklists for selective skills of first-and second-year orthopedic residents and evaluating its effects on their learning at Tabriz University of Medical Sciences.

Materials and Methods
This is a mixed (quantitative and qualitative) study that was designed and performed in Shohada Hospital of Tabriz University of Medical Sciences in 2014.We used concuss method.Our inclusion critera study is all first-and secondyear orthopedic residents.In order to design and perform DOPS tests, seven essential skills were selected by academic staff: • preparing patient on the bed of operating room for an elective orthopedic surgery • managing deep ulcer of limb • treating dislocations and fractures • all kinds of plaster cast of limbs • hand washing in operation room • debridement and stitching • using pneumatic tourniquet Evaluation checklists were prepared for each skill by taking into consideration the related literature and orthopedic faculty members' ideas.The checklists were used after determining validity.Every test was repeated three times.The content validity of the researcher-designed questionnaire was determined by experts and nine orthopedic faculty members.The Content Validity Index (CVI) was 0.95 and the test-retest method was (ICC=0.85).The reliability of the questionnaire was confirmed by Cronbach's Alpha, which was α=0.8.Faculty members' and residents' opinions were evaluated about the effects of the test on facilitating learning and skills and also helping them for preparation and success in the final Objective Structured Clinical Examination (OSCE).This questionnaire consisted of twelve closed questions and four open questions.The data were analyzed by descriptive methods of statistics (frequency, percent, mean and standard deviation).In this research, content analysis was applied manually to analyze qualitative data.Content analysis is a qualitative approach used to recognize content in data.The themes are related to backgrounds and patterns that have the features of repeatability and coding.

Ethical considerations
The research Ethics Committee of Tabriz University of Medical Sciences approved the study protocol, all the participants provided informed consent and Participation was voluntary.

Results
The mean age of faculty members was 43±5.21Four individuals were assistant faculty members, four of them were associate professors and one was a full professor.The mean years they had been working was 7.3±8.10.The mean age of residents was 28.7±1.2.According to the DOPS test results, the overall performance of residents was almost good.Also, the second testing was useful for residents with good performance, but repetition of DOPS for the third time was only useful for the residents with weak performance.We suggest the DOPS test to be repeated for the third time only for residents with weak performance results the second time.(Figure 1) In this study, we asked nine orthopedic faculty members and seven first-and second-year orthopedic residents of Tabriz University of Medical Sciences about the effects of DOPS in facilitating students' learning.Results are shown in Table 1.As Table 1 shows, most participants believed that DOPS tests can have an effective role in facilitating practiced learning and also helping them to be prepared for the final exam.Also, most of them believed that DOPS tests are effective in increasing students' procedural skills.DOPS tests provide opportunity for reflection and selfassessment, and some participants believed that DOPS tests decrease students' stress in final exams.The results of the qualitative analysis of the open questionnaire form about DOPS tests are presented in Table 2.

Discussion
•The results of a study by Bazrafkan et  for evaluating students. 13The results of a study by Bagheri et al., who investigated the effect of evaluation on emergency medical students' learning clinical skills by DOPS in Mashhad, showed that students had appropriately good performance and evaluation by DOPS had a significant effect on students' learning. 14One of the probable reasons for students' good scores in DOPS tests can be motivation that is caused by the DOPS test.The results of evaluations in different stages in this study showed that, generally, there is an increasing trend in residents' performance from the first stage to the second stage, but a decreasing trend from the second stage to the third stage.In some residents' opinions, the third test was not necessary.In a study by Akbari and Mahavelati Shamsabadi, who investigated the advantages of DOPS tests in students of restorative dentistry, 86% of students believed that two stages of tests in each period were sufficient. 15In a study by Khoshrang et al., who investigated residents' viewpoints about evaluating procedural skills by DOPS in Gilan University of Medical Sciences, every resident was evaluated four to six times in a year by DOPS and more than half of them were not satisfied with the number of times the test was held. 16In a study by Wiles et   17 In a study on surgery residents of Malaysia University, Shahid Hassan also considered DOPS tests as highly capable in promoting students' performance and learning. 18Also, Tsui et al., in their study in Taiwan, declared that these types of tests have a great role in promoting medical students' skill and competence. 19Chen et al. showed that DOPS tests in senior students of medicine In present condition, it is difficult for the tests to be useful.
They can be a good education tool besides an evaluation tool provided that it is held appropriately and principled.
It will cause boredom and dissatisfaction if it is held more.promoted self-reporting, skill and self-confidence. 20Bould et al. evaluated the validity of these tests in anesthesia and it was high as well. 23egarding the reliability of these tests, Weller et al. got high reliability for these tests. 24Other studies noted acceptable reliability too. 25,26The above problem can be reduced by teaching residents and evaluators, repeating tests and justifying learners.
•Another weak point of the present study is lack of a comparative group.We are suggesting future investigators to study more students and all procedural skills of residents.

Conclusion
Regarding the importance of clinical education, effective methods are needed in order to evaluate students' practical learning.The results of evaluating orthopedic residents in this study showed that they had appropriate performance.
According to the results of this study, repeating the tests for the second time is suitable.The strong points of DOPS tests were evaluating and promoting students' procedural skills and independence during the test, and the weak points of these tests from participants' points of view were different interactions with students and stressful tests.Considering the positive effect of DOPS in decreasing stress and increasing the results of final exams, we can modify the exam environment for improvement.Some of the most important suggestions of participants for promoting the results of DOPS tests were video recording instead of faculty members' observation, giving continuous tests, teaching the principles and methods of DOPS and informing the subject of tests before giving them.Developing and using DOPS for all clinical students is recommended and can help them to learn better and be prepared for final exams.

Study limitations
The number of residents was limited and a repetition of the study is needed for other residents or other orthopedic departments.

Figure 1 .
Figure 1.The trend of first-and second-year orthopedic residents' scores at Tabriz University of Medical Sciences on seven DOPS tests in three stages

Questions (main
themes) The results of content analysis (subsidiary themes) Strong points of DOPS tests Providing opportunity for evaluating clinical skills Self-assessing practical skills by students Recognizing students' pros and cons Students' independence and freedom at the time of exam Preparing students for final exam Providing opportunity to more communication with faculty members Weak points of DOPS tests The difference between the way and quality of evaluation among evaluators/ lack of giving equal exams for all participants Bias by evaluators Faculty members' presence and his talking to student at the time of test can be confounding.Suggestions for holding appropriate tests of DOPS Giving continuous tests along the year Teaching the principles of tests to evaluators and students before the exam Designing checklists suitable for each environment Increasing the period between tests and time of tests Recording films instead of physical presence of evaluator/faculty member Feedback of results to students besides solving students' problems Faculty members' general viewpoints about DOPS tests Education and appropriate conditions are needed for giving appropriate test.

Table 1 .
al. in Shiraz showed that 87.6% of students had an acceptable performance in DOPS tests, which are in concordance with the results of this study.Also, the researchers in that study showed that DOPS tests can be used as an effective and useful method Faculty members and first-and second-year orthopedic residents' ideas about the effects of DOPS in facilitating students' learning at Tabriz University of Medical Sciences al. in the neurology ward of Imperial College London showed almost similar results.10According

Table 2 .
Participants' points of view about DOPS test for assessing selected orthopedic skills in Shohada Hospital in Tabriz-Iran, 2013 (N=17) 11n this study, one of the advantages of DOPS tests, according to both faculty members and residents, is students' freedom and independence during evaluation.In a study by Akbari et al., 71% of students stated that DOPS tests have a positive effect on students' learning and independence and because of that they requested continuing evaluation by DOPS tests.15Also, a study by Sahebalzamani et al. showed that nurses were highly satisfied with the independence and feedback during DOPS tests.11One of the weak points of DOPS tests in the participants' points of view in this study was the unequal holding of tests for all the participants or, in other words, a lack of justice among different participants.The reason for this can be the differences between patients and different evaluators.This subject makes clear the necessity of paying attention to the reliability and validity of these types of tests.Wilkinson et al. evaluated the validity of DOPS tests and it was good.
21In addition, Moorthy et al. confirmed the validity of these types of tests in surgery.