Original Article

Using entrustable professional activity scores for ward round performance assessment of internists in internal medicine: A comparative study of two medical training models

Introduction

In Thailand, medical student recruitment and training have traditionally followed a conventional approach known as the “normal track,” which is regulated by the Ministry of Higher Education, Science, Research, and Innovation.¹ Students are admitted to medical school based on academic performance in national exams (Thai University Central Admission System). These students complete a six-year medical program, which includes three years of preclinical education and three years of clinical training. To become physicians, graduates must pass extensive exams as well as the national licensure examination. Normal track graduates are obligated to service for three years and frequently choose their workplace based on vacancy availability.² Despite attempts to boost medical graduates and reduce healthcare disparities, Thailand continues to have shortages, particularly in rural areas. To address this issue, the Collaborative Project to Increase Rural Doctor Production (CPIRD) was formed, which focuses on physician deployment in rural areas.^3,4 This project recruited students from remote areas and provided training to generate doctors who are devoted to aiding underserved populations. CPIRD students, like their conventional counterparts, must pass a national licensing exam. After finishing their first year of internship at a teaching hospital, all graduates work two years in rural hospitals to solve physician shortages.^5,6

Medical education has recently transitioned to a competency-based approach that emphasizes trainees’ ability to complete specific tasks autonomously in real-world contexts. This strategy focuses on entrustable professional activities (EPAs), which are specific tasks that trainees must complete before being allowed to practice unsupervised.^7-9 EPAs have emerged as a paradigm for assessing clinical competency, notably in internal medicine.^10,11 EPAs evaluate not just knowledge and procedural abilities, but also broad competencies including communication, teamwork, and professionalism.^12-14

This study analyzes the EPA scores of first-year postgraduate internists in internal medicine at a rural teaching hospital, with an emphasis on the standard medical training model versus the CPIRD paradigm. Its goal is to assess the effectiveness of both training programs in developing the abilities required for autonomous practice.

Methods

This was a quasi-experimental study. We conducted a comparative study of first-year postgraduate internists specializing in internal medicine during a three-month rotation of the academic year 2021 and 2023 at Sakon Nakhon Hospital, a rural teaching hospital in Thailand. The study included internists trained under two models:

1. Group 1: Conventional medical training model (Normal track)

2. Group 2: CPIRD project training model

Population

The target population for this study included all 88 first-year postgraduate internists in internal medicine who were participating in the rotation during the specified period. These students were expected to have similar foundational medical knowledge, as they were at the same point in their postgraduate training, but they would have experienced different models of training.

Sampling method

Since this is a quasi-experimental study, a non-randomized sampling method was used. The internists were not randomly assigned to the two training models. Instead, they were already enrolled in either the Conventional Medical Training Model or the CPIRD Project Training Model as part of their routine academic schedule. The inclusion of participants in each group was based on their assigned training model, and data were gathered from both groups for comparative analysis. This type of sampling method is typical in educational studies where random assignment may not be feasible due to logistical or ethical constraints. Confounding factors are variables that are related to both the independent variable (in this case, the training model) and the dependent variable (student performance), and their presence can bias the results. Control for confounding factors in this study was matching or group comparisons by ensuring that the two groups were comparable at baseline in terms of key variables. These variables might include age, prior academic performance, clinical experience, and socioeconomic status.

Sampling size

The exact number of participants in each group can be determined by looking at the total number of first-year postgraduate internists specializing in internal medicine who participated in the rotation during the three months in the academic years 2021 and 2023. The minimum sample size needed to detect meaningful differences between groups would have been determined before the study based on statistical power analysis. This ensures that the sample is large enough to yield reliable results and avoid type II errors (failing to detect a true effect). The allocated internists in this study were not randomly assigned to groups, which is characteristic of a quasi-experimental design. Instead, the students were already pre-assigned to one of the two groups based on their training model.

Data collection

The data collection was comprehensive, involving several methods to assess the students’ clinical skills and competencies during their rotation. These included structured clinical evaluations, patient records analysis, performance in simulated patient scenarios, and interviews with residents and supervising physicians. EPA categories defined by the Thai Medical Council (TMC) were used to assess competencies (Table 1), with scores assigned across five levels (Table 2).

Interviews with residents and supervising physicians provide qualitative data that can offer insights into the students’ performance and the effectiveness of the two training models. Potential interview questions were asked to gather data on various aspects of the students’ clinical skills, professionalism, and performance.

Statistical analysis

Continuous variables were presented as mean ± standard deviation and categorical variables as percentages or proportions. The Student’s t-test and Mann-Whitney test were used to compare groups. For categorical variables, the Pearson χ² test was applied. Logistic regression examined predictors of achieving above-average scores (≥ 4). Data was analyzed using Statistics Kingdom^® software (Version 2017, Australia), with significance set at P < 0.05.

Results

A total of 88 first-year postgraduate internists participated, with 64 from the conventional model and 24 from the CPIRD model. Participants’ average ages were 24.44 ± 0.687 years in the traditional group and 24.79 ± 1.693 years in the CPIRD group, with no significant gender dispersion. However, demographic data revealed significant variations in educational backgrounds between the two groups, with 78.1% of traditional trainees graduating from urban medical schools vs. none in the CPIRD group. There were also variances in terms of institution type. It was discovered that the traditional group had a higher proportion of students graduating from older institutions (Table 3). The mean total EPA scores were comparable between groups: 4.42 ± 0.498 for conventional and 4.46 ± 0.509 for CPIRD (P = 0.381). However, there were significant differences in two specific EPAs: EPA 3 (Provide proper differential diagnosis) and EPA 5 (Prescribe appropriate management), where the conventional group outperformed the CPIRD group (P = 0.036 and P = 0.034, respectively). Table 4 contains detailed information about EPA scores. There was no statistical significance found in the analysis of factors impacting the likelihood of reaching above-average total EPAs (scores ≥ 4) (See Table 5).

Discussion

This study compared the efficiency of EPAs in the training of postgraduate internal medicine interns to two training approaches: the traditional model and the CPIRD initiative. The findings revealed no significant changes in overall EPA scores between the two training groups, implying that both approaches were equally efficient at providing postgraduate interns with the necessary competencies. Both cohorts achieved equivalent levels of competency across the examined EPAs, demonstrating the effectiveness of both training methods. However, considerable variances were found in specific EPAs. Interns trained in the conventional model performed better in EPA 3 (Providing proper differential diagnosis and diagnosis) and EPA 5 (Prescribing appropriate management), although the CPIRD group seemed to do better in EPA 6 (Performing fundamental procedural skills). These distinctions were most likely due to the different emphases of the training programs. Moreover, there were no significant factors affecting the likelihood of getting above-average overall EPA scores ( ≥ 4).

Ward rounds, an important component of both hospital operations and medical education, pose significant problems for clinicians and educators.^15,16 Recent studies have highlighted their importance and the need for formal evaluation systems. Schmelter et al¹⁷ proposed an observation checklist to assess performance during internal medicine ward rounds, which was created and validated using focus group discussions and video analysis. This checklist, which included nine skills, 25 activities, and 110 observable behaviors, had strong interrater reliability, although additional validation across institutions was advised. Nørgaard et al¹⁸ examined the necessary competencies for efficient ward rounds. Based on a thorough examination of the literature and expert interviews, the study identified nine essential competency domains, including communication, collaborative clinical reasoning, and organizing. Over 70% of specialists stressed the importance of communication and teamwork, as well as self-management and mistake management, emphasizing the necessity to include these skills in medical curricula.

The majority of studies assessing internal medicine performance focused on improving EPAs. Dehghani Poudeh et al¹⁹ conducted a systematic review of various EPAs in internal medicine and classified them into six domains: care and management of the general adult population, care for patients with special needs, care coordination and communication, management and leadership, healthcare quality, education and research, and others. These areas were then separated into 14 themes and 24 sub-themes, resulting in a sophisticated framework for understanding trainee competencies. Hauer et al²⁰ assessed the pilot use of two EPAs (Discharge and Family Meeting) in an internal medicine training program. The discharge EPA received favorable feedback, with participants seeing it as beneficial for competency-based evaluation. However, opinion on the family meeting EPA was mixed, indicating that it needs additional refining to improve its utility and sustainability. Chan et al²¹ demonstrated a collaborative approach to building an EPA for transitions of care that included both residents and professors. The approach of carefully ranking 142 developmental milestones was enhanced to determine the 15 most critical milestones for care transfers. The study revealed the viability of constructing EPAs using a structured but flexible consensus-building process. Almutar et al²² examined medical students’ perceptions of ward rounds in medical and surgical settings at Kuwait University. The study discovered that students’ present competency scores were significantly lower than their expectations (P < 0.001), with bedside examination evaluated as the best-taught skill. Medical ward rounds were found to be more effective in training professional attitudes and patient interactions than surgical rounds (P < 0.001).

Overall, the lack of substantial variations in EPA performance between the conventional and CPIRD training models implies that both methods are effective in producing competency-based educational results. These findings support the continued use of both training paths, emphasizing the versatility of EPA-based assessments across a variety of training situations. However, the observed disparities across EPAs highlight the significance of adapting training programs to specific areas for growth. The higher EPA 6 scores for the CPIRD group show that rural-based training may provide more practical procedural experience, which is an important factor to consider in other training models. In contrast, the CPIRD group’s inferior performance in EPA 3 and EPA 5 may highlight areas that require more effort and resources. Recognizing these discrepancies can help influence the development of curricula and training initiatives for both traditional and rural-based programs, resulting in better overall educational outcomes.

Suggestions for curriculum adjustments that could enhance outcomes based on the two models are:

• Strengthen procedural training in conventional programs by introducing more hands-on procedural skill sessions and simulations in conventional training programs to match the practical exposure seen in CPIRD programs and partner with rural or resource-limited settings for short-term rotations to give trainees in conventional tracks direct exposure to diverse procedural cases.

• Enhance diagnostic and management training in CPIRD programs by incorporating more structured problem-based learning sessions focusing on differential diagnosis and management planning to address EPA 3 and EPA 5 gaps and leverage telemedicine or virtual case discussions with specialists from urban teaching hospitals to expand exposure to complex cases that may be less frequent in rural settings.

• Integrated collaborative training by the creation of joint workshops or case conferences where conventional and CPIRD trainees collaborate, allowing cross-learning of strengths from both programs.

• Resource allocation for CPIRD programs by increasing access to advanced diagnostic tools and decision-support systems in rural teaching settings to enhance management and diagnostic accuracy and continuing education opportunities for CPIRD trainers, focusing on the latest diagnostic and therapeutic approaches.

• Curriculum customization by tailoring specific modules to focus on commonly encountered rural challenges (for CPIRD) and urban complexities (for conventional tracks).

This study’s cross-sectional design limits its ability to provide a comprehensive picture of EPA performance. Longitudinal investigations are required to monitor the progression of clinical competence across time. Furthermore, the study’s single-center design may limit generalizability. Future studies should look at the underlying causes of variances in EPA scores, as well as the effectiveness of EPA-based training in varied healthcare settings. By identifying these areas for improvement, training schools can tailor their curricula to better educate internists for the challenges of clinical practice, thereby improving patient care in both urban and rural healthcare settings.

Conclusion

In conclusion, this study shows that both conventional and CPIRD training programs produce competent internists, as shown by comparable total EPA ratings. The found discrepancies between specific EPAs indicate the need for focused training program modifications. By using these insights, medical education stakeholders may better customize their programs to address the observed strengths and deficiencies, thereby increasing training and patient care quality.

Competing Interests

The author declared no conflict of interest.

Ethical Approval

The study was approved by the Ethics Committee of Sakon Nakhon Hospital with the number COE No. 053/2567, adhering to the Declaration of Helsinki.

References

1. Noree T, Pagaiya N, Nimnual I. Effect of doctor allocation policies on the equitable distribution of doctors in Thailand. Hum Resour Health 2023; 21(1):1. doi: 10.1186/s12960-022-00782-5 [Crossref] [ Google Scholar]
2. Witthayapipopsakul W, Cetthakrikul N, Suphanchaimat R, Noree T, Sawaengdee K. Equity of health workforce distribution in Thailand: an implication of concentration index. Risk Manag Healthc Policy 2019; 12:13-22. doi: 10.2147/rmhp.S181174 [Crossref] [ Google Scholar]
3. Pagaiya N, Kongkam L, Sriratana S. Rural retention of doctors graduating from the rural medical education project to increase rural doctors in Thailand: a cohort study. Hum Resour Health 2015; 13:10. doi: 10.1186/s12960-015-0001-y [Crossref] [ Google Scholar]
4. Techakehakij W, Arora R. Rural retention of new medical graduates from the Collaborative Project to Increase Production of Rural Doctors (CPIRD): a 12-year retrospective study. Health Policy Plan 2017; 32(6):809-15. doi: 10.1093/heapol/czx026 [Crossref] [ Google Scholar]
5. Nithiapinyasakul A, Arora R, Chamnan P. Impact of a 20-year collaborative approach to increasing the production of rural doctors in Thailand. Int J Med Educ 2016; 7:414-6. doi: 10.5116/ijme.582f.4d3b [Crossref] [ Google Scholar]
6. Arora R, Chamnan P, Nitiapinyasakul A, Lertsukprasert S. Retention of doctors in rural health services in Thailand: impact of a national collaborative approach. Rural Remote Health 2017; 17(3):4344. doi: 10.22605/rrh4344 [Crossref] [ Google Scholar]
7. Baghus A, Giroldi E, Muris J, Stiggelbout A, van de Pol M, Timmerman A. Identifying entrustable professional activities for shared decision making in postgraduate medical education: a national Delphi study. Acad Med 2021; 96(1):126-33. doi: 10.1097/acm.0000000000003618 [Crossref] [ Google Scholar]
8. Iobst WF, Sherbino J, Cate OT, Richardson DL, Dath D, Swing SR. Competency-based medical education in postgraduate medical education. Med Teach 2010; 32(8):651-6. doi: 10.3109/0142159x.2010.500709 [Crossref] [ Google Scholar]
9. Shorey S, Lau TC, Lau ST, Ang E. Entrustable professional activities in health care education: a scoping review. Med Educ 2019; 53(8):766-77. doi: 10.1111/medu.13879 [Crossref] [ Google Scholar]
10. Katz SJ, Wang D. Comparing entrustable professional activity scores given by faculty physicians and senior trainees to first-year residents. Cureus 2022; 14(6):e25798. doi: 10.7759/cureus.25798 [Crossref] [ Google Scholar]
11. Bosinski C, Rice M, Mason M, Germain LJ. Using the EPAs to evaluate the clinical experience of medical students. PRiMER 2022; 6:13. doi: 10.22454/PRiMER.2022.873914 [Crossref] [ Google Scholar]
12. Liu L, Jiang Z, Qi X, Xie A, Wu H, Cheng H. An update on current EPAs in graduate medical education: a scoping review. Med Educ Online 2021; 26(1):1981198. doi: 10.1080/10872981.2021.1981198 [Crossref] [ Google Scholar]
13. Ryan MS, Khan AR, Park YS, Chastain C, Phillipi C, Santen SA. Workplace-based entrustment scales for the core EPAs: a multisite comparison of validity evidence for two proposed instruments using structured vignettes and trained raters. Acad Med 2022; 97(4):544-51. doi: 10.1097/acm.0000000000004222 [Crossref] [ Google Scholar]
14. Valentine N, Wignes J, Benson J, Clota S, Schuwirth LW. Entrustable professional activities for workplace assessment of general practice trainees. Med J Aust 2019; 210(8):354-9. doi: 10.5694/mja2.50130 [Crossref] [ Google Scholar]
15. Ker J, Cantillon P, Ambrose L. Teaching on a ward round. BMJ 2008; 337:a1930. doi: 10.1136/bmj.a1930 [Crossref] [ Google Scholar]
16. Laskaratos FM, Wallace D, Gkotsi D, Burns A, Epstein O. The educational value of ward rounds for junior trainees. Med Educ Online 2015; 20:27559. doi: 10.3402/meo.v20.27559 [Crossref] [ Google Scholar]
17. Schmelter V, März E, Adolf C, Wölfel TL, Lottspeich C, Fischer MR. Ward rounds in internal medicine: validation of an entrustable professional activity (EPA) observation checklist. GMS J Med Educ 2018; 35(2):Doc17. doi: 10.3205/zma001164 [Crossref] [ Google Scholar]
18. Nørgaard K, Ringsted C, Dolmans D. Validation of a checklist to assess ward round performance in internal medicine. Med Educ 2004; 38(7):700-7. doi: 10.1111/j.1365-2929.2004.01840.x [Crossref] [ Google Scholar]
19. Dehghani Poudeh M, Mohammadi A, Mojtahedzadeh R, Yamani N. Entrustability levels of general internal medicine residents. BMC Med Educ 2021; 21(1):185. doi: 10.1186/s12909-021-02624-9 [Crossref] [ Google Scholar]
20. Hauer KE, Kohlwes J, Cornett P, Hollander H, Ten Cate O, Ranji SR. Identifying entrustable professional activities in internal medicine training. J Grad Med Educ 2013; 5(1):54-9. doi: 10.4300/jgme-d-12-00060.1 [Crossref] [ Google Scholar]
21. Chan B, Englander H, Kent K, Desai S, Obley A, Harmon D. Transitioning toward competency: a resident-faculty collaborative approach to developing a transitions of care EPA in an internal medicine residency program. J Grad Med Educ 2014; 6(4):760-4. doi: 10.4300/jgme-d-13-00414.1 [Crossref] [ Google Scholar]
22. Almutar S, Altourah L, Sadeq H, Karim J, Marwan Y. Medical and surgical ward rounds in teaching hospitals of Kuwait University: students’ perceptions. Adv Med Educ Pract 2013; 4:189-93. doi: 10.2147/amep.S52096 [Crossref] [ Google Scholar]