Management of Holding and Evaluating Comprehensive System of Electronic Clinical Reasoning Exams ( Sajab ) in the Sixth Nationwide Medical Sciences Students Olympiad

Introduction: The Nationwide Medical Sciences Students purpose of the Olympiad is to discover student’s talents and encourage them to study. It seems that holding regional Olympiad exams to select students for the National Olympiad can help us to maintain fairness. The aim of this study is Management of Holding and Evaluating Clinical Reasoning Exams Using a Comprehensive System of Electronic Clinical Reasoning Exams. 
Methods: Study was carried out in 2013 at the University of Medical Sciences on 750 students, 250 question designers, 37 responsibles. The nationwide universities held regional exams for the Student Olympiad in the area of clinical reasoning on specific dates and times. A quality review of the exams was done to study the strengths and weaknesses and to eliminate shortcomings and problems. Therefore, a researcher created a questionnaire with a reliability of R= 0.86 and validity was confirmed by experts, which was then loaded into the system. The collected data were analyzed using SPSS and descriptive statistics (Percent, Average, standard deviation). 
Results: The multimedia educational quality of the system, with an average of 69.36 ±22.79, the students and faculty members evaluated as good, with averages of 64.30 ±23.48 and 67.28 ± 22.43, respectively. The quality of the exam was evaluated as excellent by faculty members, with an average of 94.63 ±16.60 and 59.52 ±27.46, by the students. 
Conclusion: Evaluating the quality of the system’s performance and its ability to assess students will lead to a clarification of its strengths and weaknesses. Finally, result in the creation of a high quality evaluation system.


Introduction
The Student Scientific Olympiad was designed to identify, guide and train reasoning and problem solving in the field of health, as well as to train health providers and qualified managers to lead the health system nationally and make the direction and decisions of this system consistent with the general development direction in the country. 1 During the Olympiad, students solve problems in a competitive atmosphere; therefore, the Olympiad provides a suitable atmosphere in which to train problem-solving skills and to develop reasoning culture, creative thinking, critical thinking, teamwork and interdisciplinary activities among students. 2Achieving such goals requires fairness in the selection of students for the National Olympiad.It seems that asking the universities to contribute to questions, training students of all universities on how the academic Olympiad is held and acquainting them with various types of questions, creating a mechanism for useful cooperation between universities and holding regional Olympiad exams to select students for the National Olympiad can help us to maintain fairness.Using a comprehensive system of electronic clinical reasoning exams is one attempt to achieve the above objectives.The clinical reasoning exam is one of the most important areas of the Student Scientific Olympiad and requires attention due to its specific nature and importance in teaching reasoning skills.Clinical reasoning is of great importance in the area of health and problem solving and reasoning skills have important roles in its pedagogy. 1,2Clinical reasoning can be considered as the basis for specialists who use previous information and experiences to review and make decisions about approaching new cases.In other words, clinical reasoning is a thinking process which leads the doctor towards reasonable and purposeful diagnoses and treatments of patients, and is present in all stages of patient assessment from taking patient history to completing treatment and follow-up. 2,3Failure to properly solve clinical problems can lead both to errors in treatment and diagnostic errors, which in many cases is irrecoverable and may lead to the death of the patient. 4One of the goals of the curriculum in medicine is to create competence in the knowledge, attitude and performance of students in order to build capabilities for clinical decision making.Due to these reasons and in order to observe fairness in the selection of students for entering the national Olympiad, it is essential that the expected competencies be evaluated using appropriate and identical evaluation methods with high reliability and validity.In other words, it is necessary that the evaluation method be consistent with the created competencies and that all students be evaluated with the same methods and questions.][7][8][9] Available evidence indicates that the traditional exams (paper based) have essentially been ineffective and enjoyed low credibility due to lack of communication between different parts of patient management, including history taking, physical examination, laboratory studies, etc.10 It should be mentioned that, although unrelated, cueing by the examiner in the oral exams at the bedside and the shortcomings of the written exam are important factors in reducing the reliability of these exams. 11Due to the insufficient knowledge of some universities on appropriate methods to assess clinical reasoning skills of students, it seems that holding clinical reasoning exams at various universities with different questions causes unfairness in the selection of students for the national Olympiad.Technological developments have led to the elimination of the inappropriate cueing and shortcomings of written exams using computer simulations and, consequently, exam characteristics (reliability and validity) have improved in quality. 11The use of electronic exams for the clinical evaluation of medical students offers some advantages such as producing complex branches in problem solving, providing immediate feedback on student performance, allowing students to review the previous choices, having access to audio and video facilities during an exam, offering mechanisms to determine the exam time, controlling and monitoring the exam and providing access to the question bank to all universities which thus allows all universities to use common questions.In fact, many problems have been solved using electronic versions of clinical reasoning exams and thus these can easily be used both for training and clinical assessment.Therefore, a comprehensive system of electronic clinical reasoning exams was designed and launched to facilitate and optimize the evaluation of medical students' clinical competency; to meet the needs of faculty members, students and education professionals; to fulfill the aspirations of achieving an electronic university and to promote the overall quality of education and community health.So far, five widely-used exams to assess medical student's clinical competency were designed and uploaded within this system (Key Feature test, 5 The scenarios test, 6,7 Puzzle test, 6,8 Clinical Reasoning Problems exam or CRP test 6,9 )

Design software to set up the system
This system was designed using assumptions of INDEX and CASE models on a three-tier, object-oriented using C# programming language and Visual studio in a Net development environment and taking advantage of the SQL Server 2008 database, which can be accessed by IE / Firefox / Chrome / Safari web browsers.Obtaining a license, performance management and Education designers of questions To hold an integrated local Olympiad exam in each university in the area of clinical reasoning, after obtaining permission from the Education Deputy of Ministry of Health and Medical Education and informing universities throughout the country based on the scheduled program, an educational CD was provided containing an explanation of the design of Olympiad standard questions to acquaint the universities and question designers with the comprehensive system of electronic clinical reasoning exams (Sajab).Video conferences and question and answer meetings were held and all committee members of clinical reasoning were trained on designing questions and using the system.

Design questions and holding exams
After entering the data of 250 Olympiad question designers from all the medical universities in the country, question design began on the system.After entering questions in the 4 areas of scenario, CRP, puzzles and KF, the questions were reviewed in accordance with the Olympiad standard format and sent through the system to faculty members for feedback to make them ready for the exam after necessary modifications.Thus, 1016 Olympiad questions (410 KF, 140 scenario, 271 CRP and 195 puzzle) from other universities of medical sciences were collected in the system to hold the exam.After the exam was designed, 37 universities of medical sciences, with the participation of 750 students, held local Olympiad exams successfully using this system.It is worth mentioning samples were selected by census method.The questions were corrected and scored by the system.Thus, qualified students of every university were evaluated using Olympiad standard questions with the aim of selecting those with superior clinical reasoning to participate in the National Olympiad.

Design Questionnaire
To study the quality of holding the exam, The researchers made questionnaire include of 3 parts: 1-questions about the quality of exam (like oversight in holding the exam and designing questions, appropriate management in all of the Steps of exam, possibility of remote management,…).2questions about E-system quality (like logical function of E-system, proper graphic, the speed of options and E-pages use, the ability of circulation between the system pages, ….) and 3-questions about the quality of educational content that give to questions designers and administrators such as defined audiences goal, sound recording quality and the feasibility of system use via, mobile and tablet.In order to determine the reliability of questionnaire Cronbach's alpha (R=0.86) was used and for precise analysis of the questions of system quality assessment was given to IT professionals.The questions of exam quality and educational content placed in the hands of a few of medical education professionals survey from the viewpoint of clearness and simplicity and relatedness and necessity and the comments of all applied in questioner.

Sampling and data collection
This study was carried on 750 students, 25 question designers, 37 responsibles, in 2013 at the University of Medical Sciences.

Analysis
After collecting data from students and faculty members, these were analyzed using SPSS and descriptive statistics.

Ethical Consideration
This study does not deal with any sensitive information pertaining to human, invasive procedures and informed consent.

Results
In this study we did the evaluation using survey forms by faculty members and students in three areas: the system performance quality (Table 1, 2 & Figure 1), holding exam quality (Table 3, 4 & Figure 2), and educational content quality (Table 5) for faculty members and question designers.The results are as follows: Strengths and weaknesses and suggestions about the program by students and faculty members are expressed as follows:

Recommendation
• According to the results of holding the within university Olympiad by using the sejab system, it is recommended that using of this system in the Nationwide Medical Sciences Students Olympiad.• Considering the obtained weaknesses, the following items is recommended the necessary measures to present internet disconnection, determine the time of exam end and the suggested time to answer each question, and at the end, the better organizing of system for avoid of the some problems like hidden the some questions.• Holding a local Olympiad in each university twice a year to make students familiar with the questions.

Conclusion
The comprehensive system of electronic clinical reasoning exams was designed by the Education and Development Center at the Tabriz University of Medical Sciences.Local clinical reasoning exams were hold in universities for the first time in the Sixth Nationwide Medical Sciences Students Olympiad with the cooperation of medical universities throughout the country using this system.As has already been mentioned, the exam was done for the fair selection of students for the National Olympiad using comprehensive system of electronic clinical reasoning tests.
To be fair means the equality of opportunity for student and the interactions and behaviors and performances methods based on fairness, guidance fitting to students' abilities, and being fair in evaluating and scoring. 12ne of the objectives of using local clinical reasoning exams in the universities in the Sixth Nationwide Medical Sciences Students Olympiad using the comprehensive system of electronic clinical reasoning exams (Sajab) was to observe fairness in selecting students to participate in the national Olympiad.The local exams held in the universities previously had drawbacks, such as a lack of consistency across universities and insufficient knowledge of some universities about appropriate assessment methods of clinical reasoning skills, that led to unfairness in the selection of students for the national Olympiad.The Sajab system provides an appropriate solution by creating a large question bank, providing all universities access to appropriate questions and evaluating all students using the same questions.Another objective of this program was to develop reasoning culture Medical Students Science Olympiad has been designed to highlight and give importance to reasoning and problem solving in the health area at the University of Medical Sciences. 1 One of the main areas of health in which problem solving and reasoning play an undeniable role is clinical reasoning.Clinical reasoning has been neglected in medical education because the faculties don't take it teaching in to consideration and they are unfamiliar with appropriate evaluation methods to assess clinical reasoning skills of students. 2Four hundred faculty members cooperated to design and conduct the exam and exchanged their ideas to improve reasoning skills and assessment methods, which is of great importance as a first step towards reasoning culture development among faculty members.The other concern and shortcoming of the previous Medical Students' Olympiads was the inability of students from all universities to be exposed to the type of clinical reasoning questions in the National Olympiad, because, as mentioned, not all universities had enough familiarity with appropriate methods of testing clinical reasoning skills.The organizers of the clinical reasoning exam team have taken effective steps to hold appropriate trainings using the Sajab system to acquaint faculty members with proper assessment methods, and also provide appropriate facilities to familiarize students with the type of clinical reasoning exam questions used in the National Olympiad.Finally, despite minor shortcomings, the system has been able to solve many problems and difficulties related to the previous comprehensive exams and helps provide a more efficient, fair and better organized exam.There were not certain limitations in this study, however, holding the Olympiad tests within academic medical students across the Nationwide in 2013 for the first time in electronic form, through the system of Sjab.Because of the different ways of organizing the test as well as the different parameters measured in this study with other studies, comparison with other studies in this field does not exist.It is hoped that some efforts continue to be made to improve the quality of these exams with the cooperation and support of university authorities in the coming years.

Table 1 .
Quality assessment of electronic comprehensive system for clinical reasoning exams by students

Table 2 .
Quality assessment of electronic comprehensive system for clinical reasoning exams by faculty members Figure 1.Quality assessment of the electronic comprehensive system for clinical reasoning exams by students and faculty members

Table 3 .
Quality assessment of holding the local exam in each university in Sixth Medical Students Olympiad -clinical reasoning area by faculty members 1 Training on test method before holding the exam 66.65± 30.03 2 Easy access to the system for training 58.94±34.15 3 the test attraction 65.84±30.18 4 Enjoying the audio -video facilities during the exam 47.02±39.26

Table 4 .
Quality assessment of holding the local exam in each university in Sixth Medical Students Olympiad -clinical reasoning area by students Figure 2. Quality assessment of electronic comprehensive system for clinical reasoning exams by students and faculty members

Table 5 .
Educational content quality assessment by faculty members and question designers