Comparison of emergency medical residency pre-board and board exams among universities of medical sciences in Iran

1Department of Pediatrics, Tabriz University of Medical Sciences Tabriz University of Medical Sciences, Tabriz, Iran 2Medical Education Research Institute, Tabriz University of Medical Sciences, Tabriz University of Medical Sciences, Tabriz, Iran 3Department of Emergency Medicine, Faculty of Medicine, Tabriz University of Medical Sciences, Tabriz University of Medical Sciences, Tabriz, Iran 4Department of Neuroscience and Cognition, Faculty of Advanced Medical Sciences, Tabriz University of Medical Sciences, Tabriz, Iran 5Department of Physiology, School of Medicine, Tabriz University of Medical Sciences, Tabriz, Iran


Introduction
Testing is the most important means of measuring academic achievement. Tests that measure students' academic performance are divided into objective and subjective tests. 1 Multiple-choice questions are the most common form of questions in written exams, grouped by medical topics. 2 Like other test modes, multiple choice has advantages and disadvantages. Familiarity with these can significantly improve these tests. Among the advantages of multiple-choice tests is the fact that questions can be the same for all students, along with feasibility, high reliability, ease of grading, and objectivity. Another advantage is that multiple-choice question can evaluate a wide range of learners' knowledge in a short period. 3 On the other hand, one significant limitation is the difficulty of designing questions according to clear rules and appropriate taxonomy. 3 Although multiple choice tests are useful in examining the knowledge level of large groups of test subjects, a lack of familiarity with multiple choice design techniques Board exam, Difficulty factor, Discrimination index, Residency pre-board exam, Structural principles, Taxonomy

Original Research
significantly reduces the power and performance of this type of test. 4 The more questions are used in the design of the test, the more correct the taxonomy will be, resulting in more useful results to support the purpose of the test, which is to measure student attainment and knowledge retention. 5 One method for the preparation of theoretical questions is MCQs, for which Millman has created a checklist to evaluate question design accuracy. 6 The taxonomy of an exam question is a classification in which the level of testable knowledge is measured. Its stages are Simple Reminder (Level I), Interpretation of information (Level II), and Problem Solving and Application (Level III). In this research, questions designed in Level I are questions with low taxonomy, and questions designed with Levels II and III are questions with high taxonomy. Questions with a higher taxonomy assess more profound understanding of the topic and thus will be of greater importance in measuring knowledge. 7,8 In Iran, these tests consist of two parts: written and oral, with a four-question written test.
Many studies have been conducted in various fields of medical sciences to examine multiple-choice question: for example, in the research. There was a significant percentage of questions in the four-question Structural Error Exam, and over 90% of the questions in the current study had low cognitive taxonomy. [9][10][11] Moreover, studies have also shown that only 6.5% of the items in the Nursing School Questionnaire were designed at Level III. 12 In-country studies also show similar conditions, such that in the studies of Haghshenas and colleagues in Mazandaran, 13 Shakournia and colleagues 14 in Ahvaz a considerable number of questions (54% and 34/8% respectively) had structural problems. Shakournia et al 14 in Ahvaz and Bighlarkhani et al 15 in Hamadan found that most of the questions in the studied medical universities had lower levels of taxonomy: 89.4%, and 68.3%, respectively, and more than 75% of the questions were rated at low cognitive level.
Encyclopedia exams, and promotion of clinical disciplines in medical education residency courses, are conducted annually as a type of certification assessment used to rank and decide the promotion of assistants to higher years or to obtain a degree to serve the community. Since 2006, according to the Secretariat of the Medical and Specialty Education Council, designing and holding exams for promotion of clinical assistantships has been entrusted to type I medical universities to increase the participation of faculty members of medical departments of medical universities of the country, towards the independence of universities, the generalization of the transfer of educational powers. Thus it is necessary to review and evaluate this process (designing questions and conducting university promotion tests).
The purpose of this study was to compare items on the written promotion exams (2013 and 2014) of Emergency Medicine at the Tabriz, Tehran, Shahid Beheshti, and Mashhad Universities of Medical Sciences, with the encyclopedia exam given in the same years, and compare them with the Emergency Medicine Certificate Exam in the same year. For years, particular attention has been paid to quality improvement management and intra-university examinations.

Materials and Methods
In this cross-sectional descriptive study, the samples consisted of 600 questions from residency promotion exams and 150 questions from the board exam in 2013-2014, as well as 600 questions from residency pre-board exams and 150 questions from the board exam in the year 2014-2015 at Tabriz, Shahid Beheshti, Mashhad, and Tehran Universities of Medical Sciences. Sampling was done by a census of all questions in each year. The questions were evaluated using 12 items in Millman's checklist, [6][7][8][9][10][11][12][13][14][15][16] including stem clearness, negative options for stem, specific options, contrastive options, positive words in stem and options, writing structure of stem, duplicated options, spelling of stem and options, vertical writing of options, positivity of stem and options, and use of the "all of the above" and "none of the above" phrases in options. All questions were found to be at Levels I through III of Bloom's taxonomy and were evaluated independently by two experts in the related fields of medical education and research (including a board member of the pediatric subspecialties holding a master's degree in medical education). Guidelines were sent for each faculty member (experts) to ensure the correct understanding of the taxonomy in order to classify questions based on taxonomic definitions: Level Ι-ability to recall facts (memories), Level ΙΙ-ability to interpret information, and Level ΙΙΙ-ability to solve a new problem. 7,8 If two experts disagreed in their determination of the taxonomy level of an item, a third assessor (specialist in the field and experienced test designer) examined the same item to make a final judgment. Respecting Millman's structural principles, the project manager studied the adjusted checklist. In terms of budgeting, the chapters related to each question were determined and matched with reference books by the project manager. The discrimination index and difficulty factor of promotion questions were obtained from the Tabriz University of Medical Sciences, and the results were compared. Table 1

"Budgeting" questions in the pre-board and national board exam of emergency medicine
The number of questions per topic on the national board exam with written pre-board exams at the four universities compared, henceforth referred to as "budgeting" questions.

Emergency medicine, Shahid Beheshti Medical University
In 2013-2014, most of the questions on the pre-board exam for Emergency Medicine at Shahid Beheshti Medical University pertained to medicine and surgery topics (31, 21.1%) and trauma (50, 34%), while musculoskeletal and neurological procedures had the least (1 item, 0.7%). In 2014-2015, most questions were devoted to medical and surgical topics (45, 29.6%) and trauma (29, 19.1%) and soft tissue procedures, gastrointestinal, and vital symptoms had the least (1, 0.7%).
In 2013-2014, most of the questions on the pre-board exam for Emergency Medicine at Mashhad Medical University were on medicine and surgery topics (48, 32.0%), and trauma (23, 15.3%), and heart and specific procedures had the least (1, 0.7%). In 2014-2015, most questions were devoted to medical and surgical topics (49, 32.7%) and trauma topics (28 items, 18.7%), and evidencebased medicine and musculoskeletal and neurological procedures had the least (1, 0.7% of the total).

Emergency medicine, Tehran Medical University
In 2013-2014, most of the questions on the pre-board exam of Emergency Medicine at the Tehran University of Medical Sciences were on medicine and surgery topics (49, 32.7% of the total) and trauma (22,14.7% of the total) and evidence-based medicine and anesthesia techniques had the least (1, 0.7% of the total). In 2014-2015, most questions were devoted to medical and surgical topics (51, 34.0%) and trauma (33, 22.0%), and gastrointestinal, respiratory procedures, and musculoskeletal and neurological procedures had the least (1, 0.7% of the total).

Emergency medicine, Tabriz Medical University
In 2013, most of the questions on the pre-board exam of Emergency Medicine at Tabriz Medical University were on medical and surgical topics (49, 32.7%) and trauma (22,14.7%) and neurological and soft tissue procedures and preparedness for disasters had the least (1, 0.7%). In 2014, most items were devoted to medical and surgical topics (31, 20.7%) and trauma (24,15.8%) and neurological and soft tissue procedures, and American Heart Association (AHA) had the least (1, 0.7%).

National board exam of emergency medicine
In 2013, most of the questions on the national board exam of emergency medicine were from medicine and surgery topics (60, 39.5%) and trauma (24,15.8%) and neurological and soft tissue procedures and gynecology had the least (1, 0.7%). 1n 2014, most questions were devoted to medical and surgical topics (50, 32.9%) and trauma (23,15.1%), and neurological and soft tissue, heart, and gynecology procedures had the least (1, 0.7%).

Discrimination index and difficulty coefficient of questions in the written pre-board exam at Tabriz University of Medical Sciences
The average discrimination index and difficulty factor of the questions were examined for the written pre-board exam questions from emergency medicine at the Tabriz University of Medical Sciences in 2013-2014 and 2014-2015. In 2013, the discrimination index average was 0.18, and the mean difficulty factor was 0.64. In 2014, the discrimination index average was 0.14, and the mean difficulty factor was 0.64 ( Figure 1).

Response pattern in written pre-board exams at the Tabriz University of Medical Sciences
In 2013-2014, in pre-board exam questions from Emergency Medicine at the Tabriz University of Medical Sciences, the number of questions that had all four options selected totaled 40. These questions are more reliable than other questions in terms of design because examinees chose all four options. That is, these questions were more difficult for examinees in terms of question design. To continue, the number of questions that had three options selected was 59, and the number of questions with two options selected was 40. In 2014-2015, the numbers were similar: 39, 58, and 39, respectively.

Discussion
For this study, the researcher made a qualitative and quantitative comparison of written residency pre-board exams in Emergency  found that more than two-thirds of the questions were at Level I and only 28% were at Levels II and III. 18 In a study conducted by Sanagoo et al 19 in the nursing field, 523 questions (fewer than 5%) were at Levels II and III. However, as mentioned, this number has an upward trend.
In the present study, comparing promotion exam questions in emergency medicine at the afore-mentioned universities showed an increase in the number of questions at Levels II and III between the years 2013-2014 and 2014-2015. Questions at Levels II and III increased from 114 (76%) to 134 (89.3%) at Mashhad University of Medical Sciences; from115 (76.7%) to 130 (86.7%) at Tabriz University; from 108 (72%) to 121 (80.7%) at Tehran University; and from 120 (81.6%) to 136 (89.5%) at Shahid Beheshti University. At 13.3%, Mashhad University of Medical Sciences had the highest increase in designing more contextualized questions. In 2014-2015, the promotion exam questions, in terms of difficulty, were more similar to the national board exam in the same year: the number of questions at either Level II or III was 141 (92.8%). In this respect, the questions designed at Shahid Beheshti and Mashhad Universities of Medical Sciences were most similar to the questions in the national board exam.
In Sayyahmelli and colleagues' study 20 comparing the promotion exam questions at Tabriz University of Medical Sciences in 2000 and 2001 in internal medicine, pediatrics, general surgery and gynecology, a significant improvement in designing questions at more profound levels of learning Levels II and III was noticed, indicating improved skills of faculty members in designing questions. If exam questions only require the memorization of answer, students will be encouraged to memorize; however, if the question requires a need to interpret or solve a problem, students are encouraged to think critically and apply knowledge.
The exam questions were also examined in terms of structural problems of design. In 2013-2014 and 2014-2015, the questions on the national board exam of emergency medicine were rated at 100.0% and 99.3%, respectively, indicating little or no problems in structure. For the universities involved in this study, values ranged from 90.7% to 98.7% in 2013-2014., also indicating few problems with structure.
In Sayyahmelli and colleagues' study, 20  Khoshrang et al 21 emphasized the need for more attention to testing principles. Planning to increase the knowledge, skill, and attention of faculty members in this field can help increase this trend and improve the design of multiple-choice questions. In order to improve the current situation, it is recommended that, in addition to suitable training courses, educational groups be organized to help improve these exams through continuous assessment of residency exam questions and feedback.
In this study, pre-board exam budgeting in emergency medicine was examined at Tabriz The questions least likely to be included in both pre-board and national board exams were related to gastrointestinal, musculoskeletal, neurological, and evidence-based medical procedures. In the proposed table for budgeting pre-board exam questions of emergency medicine at Tabriz Medical University, 51 (34%) questions for medicine and surgery, and 33 (22%) questions for trauma are recommended. As mentioned, designing pre-board exam questions for the mentioned universities similar to the table has been recommended.
In the present study, the discrimination index and difficulty factor for written pre-board exam questions in emergency medicine at Tabriz  Khoshrang et al 21 noted that holding workshops has been effective in helping to design multiple-choice questions, likely due to proper design and consistency of workshop objectives as well as transparent, objective and applied use of questions designed on the residency exam for each department in order to obtain feedback during the workshop. Meyari et al 22 found that holding workshops on teaching proper design of questions accompanied by clear, objective, and applied examples was useful.
Overhauling memory-based questions is a longstanding problem in the medical education system. It can be said that this problem was seen in almost all exams at all educational levels. This situation pushes students to acquire knowledge that is shallow and easily memorized. One of the reasons for the over-use of such knowledge level questions is that such questions are efficiently designed compared to higher levels of the taxonomy. However, in recent years, a considerable number of questions have been designed at more cognitively complex levels due to the emphasis of the Council of Medical Specialties in the Ministry of Health. Efforts of the learning management system have been effective in encouraging designers to prepare questions.

Ethical approval
This article is the result of a master's thesis approved by the Department of Medical Education, which has passed the approval process according to the approvals of Tabriz University of Medical Sciences. This study did not have specific ethical considerations.