Original Article

Perception of the teachers and Phase III (part 1) MBBS students regarding objective structured practical examination (OSPE) and traditional practical examination method

Introduction

As the new medical curriculum has been introduced, now is the time to move from traditional assessment methods to a more elaborated new system of assessments. New assessment tools are designed not to assess only the knowledge part but also to assess the psychomotor and communication skills. Objective structured practical examination (OSPE) is one of the methods to minimize the variations in subjectivity, thus enhancing objectivity. Assessment tools are designed to test the attainment of educational objectives and need to be valid and reliable. Along with the knowledge domain, assessment of practical skills is a core requirement and needs to be measured reliably and uniformly with proper differentiation between various levels of performers.^1,2 An appropriate assessment of medical student’s practical competency is an integral part of the existing and the future CBME (Competency Based Medical Education) curriculum. In medical education, in the aftermath of competency-based curriculum implementation, many institutes are experimenting with an OSPE scheme though most centers are continuing with the traditional practical-based assessment. A good assessment tool is judged based on reliability, validity, feasibility, as well as acceptability.² It is well known that conventional practical examination has several problems, especially in terms of its outcome.^3,4 Traditional practical examination is more subjective and that leads to doubts over its validity and reliability while OSPE overcomes these shortcomings as it is objective and tests through direct observation assesses the knowledge, and analysis of that knowledge.

Assessment drives learning. However, to foster active learning, assessment needs to be informative.⁵ Although many options are available to do this more consistently, the OSPE is most preferred.^6,7 It involves direct observation of the students’ performance at planned stations. The OSPE can also reduce the examiners’ variability in marking the students.⁸ The new competency-based medical curriculum has specified the roles to be played by an “Indian Medical Graduate” and also described various competencies to be achieved at each level of the undergraduate curriculum. The focus is not only on “Knows” and “Knows how” but also on “Shows” and “Shows how” to achieve the required set of skills. The MCI document also emphasizes on more streamlined and continuous formative and summative assessments. The use of OSPE for formative assessment has great potential as the learners can gain insight into the elements making up their competencies as well as feedback on personal strengths and weaknesses.⁹ Due to the technicality and labor intensity, it is implemented only in a few medical colleges and universities across India. Because of the CBME curriculum on the roll, the assessment of practical skills in medical education needs to be shifted from conventional subjective methods to more objective OSPE methods.^1-3

Due to a technicality of OSPE with prior planning and coordination of the faculties, labor intensity, i.e. too many assessors are required for a simple exercise and lack of will from the faculty and management, OSPE is implemented only in a few of the medical colleges and universities in India.¹ OSPE is a specified set of tasks that assesses what students can do in a structured pattern objectively under direct observation and can assign the above-mentioned capabilities. Miller’s framework of developing clinical qualifications concentrates on four levels of assessment: “knows, knows how, shows how and does”. It was reported that OSPE evaluates the third level “shows how” of this framework by focusing on the assessment of the performance of specific skills in a controlled setting.⁴ Currently, OSPE is used for formative assessment during internal examinations in many institutes as universities have still not incorporated. This study is designed to compare OSPE over traditional Practical Examination for Phase III part 1 MBBS students. The outcome of this study will be in terms of the effectiveness of OSPE over Traditional Practical Examination and the perception of students and faculties towards OSPE in terms of feasibility, acceptability, and feasibility.

Methods

Study design

It was an analytical cross-sectional study conducted as educational research.

Study area

The study was conducted in the Department of Community Medicine, RDASMC, Ayodhya

after getting approval from the Institutional Ethics Committee.

Study population

All the available Phase III part-1 MBBS students in RDASMC, Ayodhya.

Study period

Three months.

Sample Size

92 Students, 9 faculties.

Inclusion criteria

All Phase III Part-1, MBBS students.

Study layout

After institutional ethical clearance, Phase III Part 1 MBBS students appearing for Examination in the Department of Community Medicine was selected for the study. All the students were first informed about the kind of assessment and the process of OSPE. The orientation of the entire faculty involved in the process was taken in advance, to sensitize them about the OSPE process, different stations to be made, the marking system, and checklists used at different stations. Blueprint of the checklist was validated by senior faculty members from the department. Participant students were again informed about the details of both the assessment methods, i.e., Traditional Practical Examination (TPE) and OSPE well in advance

• First, all faculty, tutors, and students were oriented for OSPE through power point presentation.

• Informed consent was taken. The orientation of the entire faculty involved in the process was taken in advance, to sensitize them about the OSPE process, different stations to be made, the marking system, and checklists used at different stations. Blueprint of the checklist was validated by senior faculty members from the department. Participant students were again informed about the details of both the assessment methods, i.e., TPE and OSPE well in advance.

• Each student has faced both methods of assessment

• Assessment was done on two consecutive days.

• Two different sets of question papers were made with the same difficulty level. After each day of examination, a feedback form will be given to participating faculty and students to assess the feasibility, acceptability, and effectiveness of the OSPE.

• The confidentiality of the stations was maintained until the end of the examination. The students underwent traditional practical examinations and OSPE.

• Feedback in the form of a questionnaire through Google form (based on 4 points Likert scale) was collected from the students about their perception of both sets of assessments. Feedback from faculty (n = 9) was also collected.

Statistical analysis

Scores were compared in terms of mean by using the Mann-Whitney U test. Descriptive tables were used to describe the values on the Likert scale regarding perceptions of faculty and students.

Results

All students were assigned to the OSPE and the same students were assigned to the traditional practical examination method. Preparation of separate mark sheets for OSPE and Traditional methods of assessment was done. Scores of each student were analyzed statistically to look for any significant difference between the two methods. Data collected from feedback forms were also analyzed statistically.

In the present study, a total of 92 students participated in the OSPE and a total of 88 students have participated in the traditional practical examination. The mean and standard deviation of marks in OSPE were 64 and 5.6 whereas in traditional practical examinations theses were 63.8 and 8.3. The range of marks obtained in OSPE was 23-90 and in TPE 42-80 (Table 1).

It can be observed in Figure 1 that a greater number (18) of students could gain marks between 75%-99% by the OSPE method in comparison to the traditional practical examination (6). While for the marks ranging between 50%-74%, the Traditional method proved to be more efficient.

Close

Figure 1.

Comparison of scores of students in OSPE and traditional practical examination

Figure 1.

Comparison of scores of students in OSPE and traditional practical examination

It is evident from the table that maximum students had scored 100% marks in the family planning method, maximum students had scored marks between 75%-99% in biomedical waste disposal while maximum students had scored marks between 50%-74% and < 50% in injection technique. Table 2 shows that at the handwashing skill station, 7 students scored 100% of marks, 51students scored marks between 75%-99%, 12 students scored marks between 50%-74% and 22 students scored < 50% of marks, and so on. At the family planning station a total of 28 students out of 92 have scored 100% marks (Table 2).

Table 3 shows the majority of students (50.9%) agree with the statement that OPSE is a fair mean of examination in comparison to traditional practical examination and are satisfied with the difficulty level (l59.6%) and believe that it is an effective and valid tool to assess knowledge (54.4%). 28.1% students strongly agreed and 36.8% of students agreed with the statement that OSPE is better than traditional Viva-voce (Table 3).

Table 3 shows that out of 92 students, 19 students were strongly agree and 49 students agreed that OSPE had covered a wide range of critical areas of the subject. Where 28 students were neutral about it, and 19 strongly disagreed. None of the students strongly disagreed with the statement.

Table 4 shows that the maximum faculty(66.7%) agree and 33.3% of faculty strongly agree that OSPE is an effective tool for assessment, is a well-organized system and covers most of the topics from the syllabus. Out of 9 faculty members, 17% strongly agree that OSPE is a better approach than the traditional method of practical examination. While 17% agreed with the statement, 17% remained neutral (Table 4).

The P value provided by the test shows that the test is insignificant indicating no significant difference between the scores of students who participated in OSPE and the Conventional method (Table 5). However, the rank sum value of the table shows a higher value for OSPE indicating that OSPE could be a better method of assessment. (Table 5).

Discussion

The NMC has implemented the new CBME curriculum for the academic year 2019–2020, where assessment is an essential step in analyzing the knowledge of the learners and acquired skills.¹⁰ This CBME curriculum has criteria for certifying the skills after assessment, the practical skills in medical education which have been followed for many years need to be shifted from conventional subjective methods to objective OSPE.¹¹ Attempts are being made to make the practical examination more reliable and valid. Few of the premier institutes in India have already started the OSPE in psychomotor skill assessment.³ This study was a small attempt to compare the traditional practical examination with OSPE in Community Medicine Department. In our study, a total of 92 students have participated in OSPE and 88 students have participated in the Traditional Practical Examination. Similar attempts have been made in different medical specialty subjects to compare the traditional methods versus OPSE.¹² A study by Trivedi et al also concluded that using OSPE as a better assessment tool with the students gives them a chance to score better.¹³ In the present study, it was found that students score higher in OSPE than in traditional methods of assessment. Another study by Nigam et al was to evaluate the efficacy of OSPE as an assessment tool compared with a conventional practical examination in the subject of community medicine with similar results.¹² The study showed students and faculties were positive towards OSPE and felt that it should be followed as a method of assessment in the practical assessment of various subjects along with the traditional Practical examination. In the present study, majority of students (50.9%) were agree with the statement that OPSE is the fair means of examination in comparison to traditional practical examination and are satisfied with the difficulty level (l59.6%) and believe that it is an effective and valid tool to assess knowledge (54.4%). 28.1% students were “strongly agree” and 36.8% of students agreed with the statement that OSPE is better than traditional Viva-voce. A study conducted by Frank J.R et al¹⁴ also reportedsimilar observations, Ananthkrishnan,¹⁵ Watson et al,¹⁶ Bairy et al,¹⁷ and Mokkapati et al¹⁸ also observed that OSPE was a well-structured, easy assessment format which the students found to be well organized, easy, and less stressful and it covered the learning domains and syllabus appropriately than conventional examination. Students were more alert during the movement around various OSPE stations and took an interest because of the division of competencies into various stations. According to our study, more than 90% of the students were satisfied with the OSPE method. The majority of the students felt that it is an unbiased and uniform method of assessment as compared to traditional practical exams. They felt that OSPE covered all the important practical questions that were the same for every student, while in TPE, there was considerable variation in the number and difficulty level of the questions asked.

Similarly in our study, OSPE was well accepted by the students as they found OSPE easy to score and less stressful than TPE. OSPE was very much appreciated by the students. The traditional practical examination has many deficiency areas where the practical skills may not be adequately observed throughout the time of practical performance. The questions are directed at the end of the practical performance. OSPE helps in improving practical skills, as students demonstrate their practical skills rather than just answering the viva. This makes it more reliable. OSPE though time and labour-intensive, can be adopted as an objective tool for the assessment of laboratory exercises because of its high reliability.

We also analyzed the perception of facultytowards the two forms of assessment as a secondary objective. The study by Radhika et al found that 94% of faculties felt that OSPE is a better method of assessment which is similar to the present study.¹⁹ The faculties have a positive attitude towards OSPE, and thus, they may be in favor of OSPE implementation as a method of assessment. The study by Mate et al found that the majority of teachers agreed that OSPE could eliminate inter-examiner bias.²⁰ The examiner’s subjectivity and favoritism are a few essential factors, which adversely affect students’ performance in traditional methods. These are minimized largely in OSPE, which help the students score better knowledge compared with Traditional examination methods.The OSPE assesses cognitive, psychomotor, and affective domains while traditional practical examination usually is useful only in determining the cognitive domain and some aspects of the psychomotor domain. The OSPE can assess the knowledge but also skills and attitudes of students in a short time. Various stations used in OSPE can be used with different portions of the syllabus, and thus students can be assessed more comprehensively.

Study limitations

In the present study, the sample size was smaller. The results of the study would be better generalized if the sample size was larger. The observations and results were based on a single time assessment by OSPE.

Conclusion

This study showed there was a significant difference in scores obtained in OSPE, in comparison with conventional practical examination. Hence, OSPE is a more effective and valid assessment tool as inter-examiner variation and bias will be eliminated. OSPE should be further enforced in other Indian medical universities and colleges. The use of OSPE as a formative tool will help in modifying teaching-learning strategies so that both the teachers and the students can derive maximum benefit. OSPE is an effective assessment tool for precisely measuring practical/clinical skills. Giving feedback to students becomes easier because of checklists.

Acknowledgments

This project was conducted as part of the ACME course. My gratitude toward the dedicated team of NMC Nodal Centre for ACME at KGMU, Lucknow for the constant support. I am grateful to all the faculty members, Tutors, and teaching support staff from the Department of Community Medicine, RDASMC, Ayodhya without whom implementation of OSPE would not have been possible.

Competing Interests

The authors declared no conflict of interest.

Ethical Approval

This study was approved by the Rajarshi Dashrath Autonomous State Medical College’s ethical committee (Ethics No. RDASMC/IEC/2023/14).

References

1. Malhotra SD, Shah KN, Patel VJ. Objective structured practical examination as a tool for the formative assessment of practical skills of undergraduate students in pharmacology. J Educ Health Promot 2013; 2:53. doi: 10.4103/2277-9531.119040 [Crossref] [ Google Scholar]
2. Pramod Kumar GN, Sentitoshi Sentitoshi, Nath D, Menezes RG, Kanchan T. Student’s perspectives on objective structured practical examination (OSPE) in forensic medicine - a report from India. J Forensic Leg Med 2015; 32:39-41. doi: 10.1016/j.jflm.2015.02.013 [Crossref] [ Google Scholar]
3. Bhat VS, Kishan Prasad HL, Bhat SP. A study of effectiveness of OSCE in the formative and summative assessment of medical students - a student-centred study. Int Educ Res J 2016; 2(3):25. [ Google Scholar]
4. Miller GE. The assessment of clinical skills/competence/performance. Acad Med 1990; 65(9 Suppl):S63-7. doi: 10.1097/00001888-199009000-00045 [Crossref] [ Google Scholar]
5. Gupta P, Dewan P, Singh T. Objective structured clinical examination (OSCE) revisited. Indian Pediatr 2010; 47(11):911-20. doi: 10.1007/s13312-010-0155-6 [Crossref] [ Google Scholar]
6. Harden RM, Gleeson FA. Assessment of clinical competence using an objective structured clinical examination (OSCE). Med Educ 1979; 13(1):41-54. [ Google Scholar]
7. Bloom BS. Taxonomy of Educational Objectives: The Classification of Educational Goals. Chicago: Susan Fauer Company Inc; 1956. p. 201-7.
8. Wani P, Dalvi V. Objective structured practical examination vs traditional clinical examination in human physiology: student’s perception. Int J Med Sci Public Health 2013; 2(3):543-7. doi: 10.5455/ijmsph.2013.080320133 [Crossref] [ Google Scholar]
9. Shenwai MR, Mardikar P, Joshi N, Joshi RN. Introducing objective structured practical examination as a formative assessment tool for phase I medical professionals in physiology. Natl J Physiol Pharm Pharmacol 2020; 10(8):619-24. doi: 10.5455/njppp.2020.10.04094202001052020 [Crossref] [ Google Scholar]
10. Lockyer J, Carraccio C, Chan MK, Hart D, Smee S, Touchie C. Core principles of assessment in competency-based medical education. Med Teach 2017; 39(6):609-16. doi: 10.1080/0142159x.2017.1315082 [Crossref] [ Google Scholar]
11. Pramod Kumar GN, Sentitoshi Sentitoshi, Nath D, Menezes RG, Kanchan T. Student’s perspectives on objective structured practical examination (OSPE) in forensic medicine - a report from India. J Forensic Leg Med 2015; 32:39-41. doi: 10.1016/j.jflm.2015.02.013 [Crossref] [ Google Scholar]
12. Nigam R, Mahawar P. Critical analysis of performance of MBBS students using OSPE & TDPE - a comparative study. Natl J Commun Med 2011; 2(3):322-4. [ Google Scholar]
13. Trivedi RS, Diwan JS, Shah CJ, Jani RD, Anand AK. The influence of objectively structured practical examination (OSPE) on scoring pattern. Int J Biomed Adv Res 2014; 5(2):87-9. doi: 10.7439/ijbar.v5i2.632 [Crossref] [ Google Scholar]
14. Frank JR, Snell LS, Cate OT, Holmboe ES, Carraccio C. Swing SR. Competency-based medical education: theory to practice. Med Teach. 2010 32, 638–45.
15. Ananthakrishnan N. Objective structured clinical/practical examination (OSCE/OSPE). J Postgrad Med 1993; 39(2):82-4. [ Google Scholar]
16. Watson AR, Houston IB, Close GC. Evaluation of an objective structured clinical examination. Arch Dis Child 1982; 57(5):390-2. doi: 10.1136/adc.57.5.390 [Crossref] [ Google Scholar]
17. Bairy KL, Adiga S, Shenoy S, Magazine BC, Amberkar M, Meena Kumari K. OSPE in pharmacology: comparison with the conventional method and students’ perspective towards OSPE. Br Biomed Bull 2014; 2(1):218-22. [ Google Scholar]
18. Mokkapati A, Pavani G, Dass SM, Rao MS. Objective structured practical examination as a formative assessment tool for 2nd MBBS microbiology students. Int J Res Med Sci 2016; 4(10):4535-40. [ Google Scholar]
19. Radhika G, Dara AK, Varalaxmi KP, Bhavani C. Perceptions of the introduction of objective structured practical examination (OSPE)/objective structured clinical examination (OSCE): a pilot study carried out in Government Medical College, Ananthapuramu, Andhra Pradesh, India. J NTR Univ Health Sci 2015; 4(3):145-9. doi: 10.4103/2277-8632.165401 [Crossref] [ Google Scholar]
20. Mate VH, Sahasrabudhe RA, Pandit VA, Beri SG. OSPE as the tool to assess application of pharmacology knowledge of II MBBS students: teachers’ perceptions. Int J Health Sci Res 2016; 6(6):117-22. [ Google Scholar]