Medical students’ evaluation of competency-based and traditional educational environment using the Dundee Ready Education Environment Measure

Background: Competency-based medical education (CBME) was introduced in India in 2019 to transform the educational environment. Students’ perspectives are vital to incorporate positive changes and ameliorate shortcomings as important stakeholders. The authors have tried to capture these perceptions in the study. Methods: A cross-sectional study was conducted between August 2020 and May 2021 with randomly selected undergraduate student volunteers at a government medical college in Delhi, India. In all, 50 students were included from the first year enrolled in a competency-based curriculum and 50 from the second year studying a traditional curriculum. The Dundee Ready Education Environment Measure (DREEM) was used to evaluate the perceptions of the educational environments. Results : Perception of learning (PL) showed a more positive response among the first-year students (using CBME) with a mean score of 32.18 ± 6.32 in comparison to the second-year students following the traditional curriculum with a mean score of 29.04 ± 7.29 ( P = 0.04). Total score, PL, students’ perception of teachers/teaching (PT), and academic self-perception (ASP) ( P = 0.03, < 0.01, 0.02, 0.05 respectively), were higher in day-scholars in comparison to those who stayed in college-based housing (hostellers). Overall, no gender differences were seen. Conclusion : Better PL, a reduction in problem areas, and a favorable environment in day scholars compared to hostellers were all seen in students following the CBME method. Perceptions of the two groups of students concerning support systems and other aspects such as students irritating teachers, cheating, and perceived boredom were different. This analysis of the educational environment can serve as helpful feedback to curriculum designers.


Introduction
There has been an increasing realization around the world of a gap between the training of medical graduates from a traditional structured (TS) medical education system and society's needs. The consequent move towards redesigning teaching and training has led to the formulation of a competency-based medical education (CBME) curriculum. CBME is known for its unique characteristics of being student-centered and outcome-oriented, with assessment mimicking real professional tasks and content responsibility shared between student and teacher. A recent review of CBME describes what it is, how it is different from the traditional structure and process-based education, shortcomings of the conventional system, and the need for change. 1 The benefits of this educational system for both students and faculty and the challenges associated with its teaching and assessment modalities are manifold. 2 The Medical Council of India (MCI; now the National Medical Commission), the regulatory body for medical education in India, introduced CBME to the undergraduate medical curriculum with the entering class of 2019 to replace the TS curriculum that has been followed for many decades.
The formulation of CBME is based on the perception of what is essential for an Indian Medical Graduate (IMG) to become a competent doctor. 3 The underlying premise is that the student realizes the importance of the knowledge and skills acquired during undergraduate medical education and accesses them easily when presented with an actual clinical situation. Horizontal and vertical integration involves correlating subjects across different professional years and provides students with a broader perspective. Many of these practices existed in TS; however, CBME provides a more conscious and consistent emphasis to these elements. For instance, clinical postings started from the second professional year in the TS curriculum, while the CBME curriculum ensures that first-year professional students also gain this experience through Early Clinical Exposure (ECE) . 4 To impart proper communication skills and maintain patient-confidentiality and build mutual trust and empathy, CBME introduced an Attitude, Ethics and Communication (AETCOM) Module for social and psychological aspects of patient care. 5 Hence with the introduction of CBME, the two-way process of the educational environment has undergone a major change for teachers as well as for students.
The Dundee Ready Education Environment Measure (DREEM) is often used to study the educational environment in many national and international studies. 6,7 The educational environment is one of the determinants that help implement a curriculum successfully. 8 Since the CBME curriculum was only introduced two years ago, evaluation reports assessing the effects of the new curriculum are not available. Data around student performance are also not sufficient to quantitatively compare the benefits of the new curriculum to TS. Therefore, this study aimed to evaluate the educational environment of both CBME and TS by soliciting feedback from the students using a DREEM questionnaire to perform a detailed analysis of these perceptions and thus provide a mirror of its strengths and shortcomings. It is hoped that the conclusions and suggestions generated by this study will be implemented to make CBME even more relevant to building a genuinely competent and complete IMG in the coming years.

Participants in the study
This cross-sectional study encompassed two undergraduate student study groups (N = 100): a first-year group (mean age = 20 years; n = 50) and a second-year group (mean age = 20.08 years; n = 50) at a government medical college in Delhi. The distribution of the 50 students per year as male: female was 28:22 for the first year and 26:24 for the second year. The first-year students (admitted to the medical college in 2019) were the first batch to encounter the CBME curriculum, whereas the second-year students (admitted to the medical college in the year 2018) were the last batch to follow the TS curriculum.

The instrument for data collection
A pre-validated questionnaire (DREEM) was used to collect data from the study participants. 9 This is considered a valid and reliable tool in Indian education, as English is the official language of instruction in Indian medical colleges. 10 It consists of 50 statements using a 5-point Likert-type scale ranging from Strongly Agree (4) to Strongly Disagree (0). Of these 50 items, nine (4,8,9,17,25,35,39,48,50) are negative; thus, the scoring is reversed for these statements, and a more negative score indicates higher agreement. As is evident from the above point system, the maximum possible total for the questionnaire is 200 points. This questionnaire includes 5 subscales: Perception of learning (PL), consisting of 12 items with a maximum total of 48 points; students' perception of teachers/teaching (PT) with 11 items and a maximum of 44 points; Academic self-perception (ASP) with 8 items and a maximum of 32 points; Perception of the atmosphere (PA) with 12 items and a maximum of 48 points; and social self-perceptions (SSPs) with 7 items and a maximum of 28 points.
Permission to use the DREEM questionnaire for the study was obtained from the authors via email.

Methodology
The study was conducted from August 2020 to May 2021 and assessed the educational environment between August 2019 and March 2020. This study measured the educational environment before the COVID-19 pandemic began in India. First, approval from the medical college's Institutional Ethics Committee (IEC) was obtained. Sample size was calculated based on the assumption of an anticipated standard size effect of the perception score between the two groups as 0.60. Two-sided alpha is 0.05, power 80% and using the formula n = 2(Z 1 -α/2 + Z 1β) 2 /effect size 2 where Z 1 -α/2 = Value of Z in standard normal curve beyond which area is 0.025 (= α/2) and Z 1 -β = Value of Z in standard normal curve beyond which area is 0.20 (= β). The required number of participants in each of the two groups using this formula was 44. Taking into consideration a 10% non-response estimate, 50 participants were selected from each of the two batches. Each batch consisted of a total of approximately 250 students.
All students were included in the population group. Study participants were randomly selected using a random number table. Selected students were contacted by telephone, WhatsApp and/or email asking for their willingness to participate in the study. If the student did not wish to participate in the study, the next one in the random number table was contacted. The study participants were sent a Google form link which contained the participant information sheet and the informed consent form, which was mandatory to be signed and submitted with volunteer details. The participant information sheet contained relevant information regarding the study, which their class representatives shared with the two cohorts. The participants were assured that their responses would be confidential in the event of publication and would only be visible to the investigators. Once volunteers had submitted consent forms and relevant details, they were sent a Google form link containing the DREEM questionnaire to their respective e-mail IDs. They were asked to fill out the questionnaire within two days, and the whole procedure was completed in seven days.

Data entry and statistical analysis
Participant data were coded, scored, and analyzed using SPSS 21 statistical analysis software. Quantitative data were expressed using mean ± standard deviation and, based on normality distribution difference between two comparable groups, either Student's t test (unpaired) or Mann-Whitney U test was used. A P value of less than 0.05 is considered statistically significant.

Results
The details of the scores are shown in Table 1. Students reported similar scores in both years, although minor variations in the percentages were seen. The total score indicated that perception was more positive than negative for both years. Looking at the subscales for both years, PL showed ' A more positive approach' , PT showed 'Moving in the right direction, ASP suggested that the students were 'Feeling more on the positive side' , PA reflected ' A more positive atmosphere' and SSP was perceived as 'Not too bad. ' PL was higher in the first year. Rest there were no significant differences between the two groups.
Significant differences were seen in some of the subscales item numbers, as shown in Table 2. Items 9, 17, 25, 39, and 50 are negative statements; thus, a low score indicates the students agreed with the statement. Problem areas were identified by scores ≤ 2, which were more prevalent among the second-year students. The comparison with the values of first-year students following CBME is shown in Table 3. Both years showed lower scores for items 25, 27, and 50: overemphasis on factual learning, the ability of students to memorize, and students irritating the teacher were areas of concern for both groups of students. In addition, second-year students disagreed that there was a good support system. They were rarely bored in the course, did feel cheating was a problem, and that teachers were MSP, Maximum score possible; FS, first-year score; FPTS, first year % of the total score; FRS, first year range of score; FMSSR, first-year maximum student score range; PSFMSSR, % students in a first-year maximum score range; SS, second year score; SPTS, second year % of total score; SRS, second-year range of Score; SMSSR, second-year maximum student score range; PSSMSSR, % students in second-year maximum score range; PL, Perception of learning; PT, Perception of teaching; ASP, Academic self-perception; PA, Perception of atmosphere; SSP, Social self-perception. % of total indicates score as a percentage of the total possible score P value shows the comparison of the first and the second-year score. P < 0.05 is significant authoritarian and got angry in class. Only item number 2 scored ≥ 3.5 (among first-year students), suggesting improvement is needed in most domains.

Gender differences observed in the two years of students' perception
There were no gender differences in overall perception. In the first year, female students recorded a higher total as well as subgroup score but an opposite trend was seen in the responses of second-year students, for the total as well as all subgroups scores except SSP, and the differences were not significant. Some variations in responses to specific questions were seen in respondents of both years. For the first-year students, significant variations were seen in item number 35 (2.57 ± 0.790 and 3.05 ± 0.722 for males and female students, respectively (P = 0.02)), suggesting that male students in the first year felt significantly more disappointed by the atmosphere. For the second year PL item 47, the score was higher for male students, 2.62 ± 1.061 in comparison to1.96 ± 1.16 (P = 0.04) in females, who perceived that 'long term learning is not emphasized' as a problem. Similarly, PT item 32 (teachers providing constructive criticism) had results of 2.54 ± 0.706 and 2.04 ± 0.955 (P = 0.02) for male and female students, respectively. SSP item 28 (I seldom feel lonely) was 1.92 ± 1.093 and 2.54 ± 1.021 (P = 0.04), where male students thought loneliness was significantly more of a problem.
To understand whether differences were related to the year or gender or both, male students' responses in the first and second years were compared; no significant differences in total or sub-domain scores were seen. Individual differences in items are shown in Table 4.
The total score of female students is shown for the first and second years, respectively: 134.23 ± 16.18 and 122.08 ± 22.54 (P = 0.05); PL was 32.86 ± 4.32 and 28.25 ± 7.30 (P = 0.01). The individual differences in items are shown in Table 4.

Comparison between 'day scholars' and 'hostellers'
Day scholars are students who do not stay in universityprovided accommodations and usually commute from home every day. Hostellers are those who stay in universityprovided accommodation along with fellow students. In first-year students, a significant difference was seen in day scholar and hostellers between Total, PL, PT, and ASP, with lower scores seen among hostellers (Table 5). No difference was observed in second-year students. Comparison between subscale scores of day scholars in both first and second years showed significant differences in PL scores: 34.46 ± 6.41 (first year) and 29.65 ± 7.15 (second year) (P = 0.01) and in PT scores: 30.67 ± 4.79 (first) and 26.25 ± 6.38 (second) (P = 0.02). PL/Total % was 71.78 ± 13.36 and 61.77 ± 14.90 (P = 0.01), respectively; PT/Total% was 69.69 ± 10.88 and 59.66 ± 14.50 (P = 0.02), respectively, for the first and second years. Significant differences between individual item numbers in subscales are shown in Table 6.
Comparison between items in the subscale of hostellers in their first and second years are outlined below. In all cases, the first number is for first-year students, and the second is for second-year students. Results showed that ASP item 10 (confidence about passing the year) was 2.85 ± 0.88 and3.27 ± 0.74, P = 0.04; ASP item 26 (last year's work being a good preparation of this year's work) was 2.35 ± 0.56 and 2.70 ± 0.65, P = 0.01. PA item 17 (cheating) was a problem for second years, 2.19 ± 1.13 and1.50 ± 1.07, P = 0.02. ASP item 33 (Socially comfortable) was 2.65 ± 0.69 and 2.97 ± 0.81, P = 0.04. For SSP, item 14 (related to boredom) was 2.19 ± 0.94 and 1.53 ± 1.19, P = 0.02, signifying a problem for second-year students.

Discussion
After introducing CBME, it is natural to expect differences in the perceptions of the educational environment. Although DREEM has been used to evaluate the educational environment in many studies as mentioned above, its use is a new aspect of this study to evaluate CBME by comparing it with TS along with the comparative perception of day scholars and students staying in hostels who spend relatively more time in their educational environment.
The students' total scores in both years were between 100-150 out of 200. This means that their perception was more positive than negative for both groups. Two MSP, Maximum score possible; FDS, First-year day scholar score; FDPTS, first year day scholar % of the total score; FDRS, first-year day scholar range of score; FHS, First year hosteller score; FHPTS, first year hosteller % of the total score; FHRS, first-year hosteller range of score; PL, Perception of learning; PT, Perception of teaching; ASP, Academic self-perception; PA, Perception of atmosphere; SSP, Social self-perception. P value shows the comparison of the first-year day scholar and hosteller score. *P value for comparison of FDPTS and FHPTS. P < 0.05 is significant. other studies from India have reported similar results in medical students. 10,11 The total score of first-year students in our study was higher than second years, though the difference was not significant. In an Indian study, the total DREEM score was lower for the first-year than secondyear students. They found significant differences in the total and subscale scores between the years. 12 In a Korean study, the score was lowest for the first-year students and highest for senior years. 13 These observations are in contrast to our study. We found that the PL was higher in the CBME group, although it was the first-year group. It is reasonable to expect that because the CBME curriculum provides more versatile learning, enabling students to correlate with clinical subjects right from their first year due to ECE, the subject matter becomes more interesting and easier to retain, resulting in a higher perception of the educational environment. According to the DREEM questionnaire, an average item score of ≤ 2 out of the maximum possible score of 4 denotes a problem. This study found that the CBME curriculum had 3 problem areas compared to the 8 seen in the TS curriculum and no new problem areas emerged. In common problem areas, namely, factual learning, memorizing all teaching material, and students irritating teachers, the CBME curriculum scored higher, signifying a better educational environment. The problems of secondyear students (following TS), were related to authoritarian, angry teachers, boredom, cheating, and a lack of support system. Memorizing of facts as reported by our students and lack of support system when students are stressed, as pointed out by our second-year students, has also been reported in a study of third-year students, whose DREEM scores were in the range of 130-135. This study compared scores with different schools around the world. 14 Similar problem areas such as student boredom, creative and teacher-centered teaching, emphasis on factual learning, and lack of support system were observed for TS curricula in various colleges across India, 10,11 The problem area of cheating was also reported by fifth-year students in Saudi Arabia. 15 Other studies have also identified similar problem areas such as overemphasis on factual learning, boredom, overreliance on memorizing everything, and authoritarian teachers, indicating that problem areas are generally related to the TS curriculum and not specific to any institution. 16 These observations also suggest that problems of senior students, whether it is the second or fifth year, may be similar.
We did not find any overall significant gender differences in our study. This is similar to the above-mentioned study at KSU. 15 Other studies have reported differences between genders 10 although both higher scores in males 13,17 and females have been reported. 6 The first-year day scholars' scores were statistically higher for the total score as well as subgroup scores of PL, PT, and ASP compared to hostellers. Comparing day scholars of both years showed that all first years' scores were higher than second years, although differences existed for some items. This suggests that medical students may perceive the educational environment based on how much time they spend on campus, and there could be some differences in perception based on gender, which may not be substantial enough to appear as a significant overall difference. A study in Sudan showed that students who stayed with their families scored higher than those who stayed in rented apartments or with relatives. 17 There have been studies that have found no difference between perceptions of hostellers and day scholars. However, they did not state the type of environment they have evaluated. 18,19 A recent study on various aspects of CBME, including a foundation course, integration, electives, and assessment, used a Likert scaled questionnaire and studied first-year students, but did not study the educational environment. 18 Studies on CBME evaluation have generally used Likert scaled individual questionnaires instead of DREEM. 21,22 Both a Nigerian 23 and a Sri Lankan study 24 have evaluated the response to a change in the curriculum which has become learner-centered. Faculty perceptions must also be solicited, and in one study, the lecturers' perceptions were similar to those of the students except for the indicator for criterion-referenced assessment of CBME. 25 Therefore, evaluation of the competency-based teaching using DREEM is a new concept in our study, which has added to the existing knowledge and shows a difference in perceptions of CBME and TS educational environments. It has also looked into the views of students who stayed in the hostel, thus spending more time in the environment, and compared their responses to the day scholars.

Lacunae/Limitations of the study
The students, course content, teachers' experience on the curriculum, and the lockdown under the pandemic are other factors that could have contributed to the differences observed.

Scope for future
Analyses of perceptions of the faculty and other stakeholders of CBME should also be conducted.

Conclusion
A better perception of the educational environment was seen in students following the CBME curriculum. They felt that the teachers encouraged active participation, provided constructive criticism, and helped them build confidence and that their time was put to good use. The response was higher among day scholars than hostellers in the first year. There were differences in perceptions among the two years concerning the support system, students irritating teachers, cheating, and boredom. Gender differences were not observed overall but were seen in some questionnaire items. Problem areas were identified in the environment related to both types of the curriculum but were lower in CBME. Some CBME specific problems overemphasized factual learning and memorizing content, which should be addressed to make it more effective and acceptable. This analysis of the educational environment can serve as feedback to curriculum designers to improve learning.