Introduction

The concept of competency-based medical education (CBME) has gained worldwide recognition as a driving force for innovation in medical training curricula, in both undergraduate and postgraduate (PG) medical education ( 1 ). CBME reflects a paradigm shift from time-bound to outcome-driven and learner-centered framework ( 2 ). The concept of Entrustable Professional Activities (EPAs) was introduced to enable the implementation of CBME, with the aim of guidance and assessment of learners combined with patient care in clinical settings ( 3 ).

Operational Definition of EPA: As introduced and defined by Olle ten Cate, an EPA is “a unit of professional practice that can be fully entrusted to a trainee, as soon as he or she has demonstrated the necessary competence to execute this activity unsupervised” ( 3 ). The EPAs framework facilitates clinical educators and learners in progressive attainment of their autonomy and responsibility that ultimately prepares them for unsupervised practice of essential tasks in their professional practice ( 4 ).

The EPA-based assessment is structured to delegate critical tasks while maintaining a predetermined level of supervision. The assessment of the trainee mainly aims to measure the extent of supervision necessary for an EPA, as per the five designated EPA levels ( 3 , 4 ).

Various accrediting bodies, including the Canadian Medical Education Directives for Specialists (CANMEDS) and Accreditation Council for Graduate Medical Education (ACGME) have successfully developed and implemented a competency-based framework and very recently the EPAs ( 2 , 5 ). Academic institutions across the globe that have incorporated Entrustable Professional Activities (EPAs) into undergraduate and postgraduate medical education have systematically adapted and validated these activities to reflect their cultural frameworks, healthcare delivery models, and the specific needs of their patient populations ( 6 ).

While EPAs have been established for several medical subspecialties, there is a dearth of literature on their application to pathology residency, especially in hematology training ( 7 ). It is important to note that a uniform set of core EPAs cannot be universally implemented without due consideration of local contexts and healthcare systems. Medical institutions planning to integrate EPAs into their curricula must undertake their own EPA development process that considers their country's unique circumstances. The limited EPA research in Asia, particularly in low- and middle-income countries, highlights the need for studies that consider cultural and regional variations in the development or implementation of EPA ( 5 ). In 2020, the Department of Anaesthesiology at Aga Khan University, Pakistan, conducted a study to reform its postgraduate training program by implementing competency-based medical education, integrating simulation-based teaching, and utilizing assessment tools such as entrustable professional activities and daily clinical evaluations to enhance educational outcomes ( 8 ). Moreover, there is a scarcity of evidence on expert consensus regarding the identification of core EPAs and corresponding competencies for postgraduate hematology residency programs in a low-resource setting such as Pakistan. Moreover, implementing EPAs is hindered by cultural resistance to new teaching methods, a lack of supportive policies, and resource constraints ( 9 ). The research question was “What are the essential EPAs in hematology and what should be their teaching, learning, and assessment strategies as per the consensus of the experts.” The objective of the study was to develop essential EPAs and their respective competencies, teaching, and assessment strategies for hematology residency programs in Pakistan, through expert consensus using a modified Delphi method.

Methods

The study was conducted at the Section of Hematology and Transfusion Medicine, Department of Pathology and Laboratory Medicine, Aga Khan University Hospital (AKUH), Pakistan.

The Aga Khan University Hospital (AKUH) in Pakistan has evolved into a global institution with a strong focus on improving healthcare standards and addressing healthcare needs in the developing world ( 10 ). The College of Physicians and Surgeons in Pakistan (CPSP) is primarily responsible for postgraduate medical education and training. It offers Fellowship training in 88 specialties and Membership in 22 various specialties, which are among the highest levels of certification for medical professionals in Pakistan. The CPSP offers two Hematology training pathways: 1) Hematology, for candidates who have passed the Fellow of College of Physicians and Surgeons (FCPS) Part I written exam in Pathology and complete 4 years of specialized training in Laboratory Hematology, and 2) Clinical Hematology, for Fellows (FCPS Part II cleared) in Medicine, Pediatrics, or Medical Oncology, or those who have passed the FCPS Part I in Medicine and Allied fields and completed a two year training, i.e. Intermediate Module (IMM) in the mentioned specialties. The Postgraduate Medical Education (PGME) program at AKUH has been a pioneer in the field of medical education in Pakistan, with AKUH’s systems and processes influencing PGME across the country. The PGME programs offered at AKUH include residency programs in 34 specialties, recognized by CPSP, along with a 1-year internship and various fellowship programs ( 10 ). This also includes residency programs in Pathology across four specialties: 1) Hematology, 2) Histopathology, 3) Chemical Pathology, and 4) Microbiology. Recently, the PGME department and its five residency programs at AKUH in Karachi, Pakistan, have been accredited by the ACGME-International, making it the first institution in South Asia to receive this benchmark recognition. Competency-based curricula are being developed and implemented for all postgraduate programs at AKUH as a pioneer and being the first of its kind in Pakistan ( 10 ). Hence, the residency training in Hematology, offered by the Section of Hematology, Department of Pathology and Laboratory Medicine at AKUH, is not an exception.

Identification and elaboration of EPAs

Before the development of the questionnaire and Delphi study, 16 essential EPAs were identified and elaborated by a group of in-house hematology faculty at the Department of Pathology and Laboratory Medicine, AKUH, with group consensus. Each group member performed an extensive literature search and reviewed the available relevant articles regarding competency-based assessments and entrustable professional activities in hematology. An extensive evidence-based literature review, faculty experience, and expertise were utilized to identify these most specific and essential tasks related to hematology postgraduate education, which could be described as EPAs. Each EPA was elaborated into specific requirements and components of knowledge, skill, and attitude (KSA). These components were then matched with the six core ACGME competencies of medical knowledge (MK), patient care (PC), interpersonal and communication skills (IP&CS), practice-based learning and improvement (PBL&I), professionalism (P), and systems-based practice (SBP) ( 11 ). The level of supervision for EPAs was set to Level 4, where residents from PG years 1-4 will be allowed to practice unsupervised during their postgraduation residency training as specified ( 4 ). Additionally, multiple teaching/learning and assessment strategies for each of these EPAs were also identified. The EPAs were reviewed by four content specialists and two medical educationists for content validation and to ensure that the desired attributes of EPAs were met.

Study Design

The modified Delphi Rounds

For this cross-sectional study, a modified Delphi method was employed as an expert consensus approach, which consisted of two iterative rounds. The Delphi study is a structured and anonymous iterative process that involves collecting judgments from experts, using a series of questionnaires with controlled feedback until a group consensus is reached ( 12 ) A predefined criteria for consensus of EPA was set to conclude where at least 80% agreement was required from the participants on the top two measures (i.e., "extremely important" or "very important"), median≥4, inter-quartile range (IQR)≤1 ( 12 , 13 ). For teaching/learning and assessment strategies consensus was defined as ≥80% agreement as selection.

Selection of Experts

To account for potential dropouts in the rounds, a total of 22 experts were invited using purposive sampling techniques to enter the study by an individual email or a phone call; out of them, 16 experts across 13 institutions agreed to participate and were included in the study. The experts were identified as Hematology faculty involved in the Hematology residency program at multiple institutions nationwide. They were defined as consultant hematologists as a Fellow of College of Physicians and Surgeons (Pakistan) and /or Fellow of Royal College of Pathology (UK) or with equivalent qualification, involved in the teaching and assessment of postgraduate (residency) training in Hematology, with a minimum of two years of teaching/academic experience.

Data Collection

The data for this study was collected using an online Google form, and the participation of the experts was entirely voluntary and with their consent. The participants were invited via email, which provided them with a detailed explanation of the study's purpose and the online questionnaire link. An online questionnaire was developed seeking some demographic details and incorporating all identified EPAs and their components, such as knowledge/skill statements, competencies, teaching and learning, and assessment strategies. The participants were expected to respond using a 5-point Likert scale (ranging from 1= "not at all important," to 5= "extremely important") for EPAs and select educational and assessment strategies for their consensus. In the first round of Delphi, each item of EPA and its elaboration and components on the questionnaire were followed by an additional open-ended question, allowing the experts to add any suggestions or new items if required. Similar methodology was used in the second round to seek consensus on any newly suggested items by the panelists or on the items/components that did not reach consensus in the first round. Data were anonymized, secured, and was only accessible to the investigators, with the confidentiality of participants assured. The study entire methodology is displayed in the flowchart (Figure 1).

Figure 1. Flow chart showing the complete study process

Data Analysis

The data were analyzed using IBM Statistical Package for the Social Sciences (SPSS) version 19 to determine the frequencies and percentage of agreement, and STATA (STATA 17, StataCorp, College Station, TX) was used to determine the mean and SD values. Cronbach’s alpha was determined for reliability, and Wilcoxon rank-sum and independent t test were used to determine the response stability of the Delphi rounds.

Ethical considerations and consent to participate

Ethical approval was obtained from the institutional ethical review committee (ERC # 2022-7797-23361) of Aga Khan University before starting the study. Informed consent was taken from all participants before enrollment in the study.

Results

A total of 16 experts across 13 national institutions participated in the first round of Delphi with a response rate of 72.7%. The participants included 10 (62.5%) females and 6 (37.5%) males, as shown in Table 1, which also shows other demographics. The participants were representatives of various institutions from all major cities of three provinces across the country and included 14 FCPS, 1 Diplomat of the American Board of Pathology, and 1 FRCPath. Two experts were additionally qualified as Fellows of the American College of Physicians (FACP) and master’s in health Professions Education (MHPE), respectively. The mean work experience of expert panelists as the faculty was 9.81 years±6.5 SD. Ten experts had a mean experience of 4.94 years±7.2 SD as residency program directors (PD). The Cronbach’s Alpha of the questionnaire was found to be 0.85.

Delphi Round 1

In the first round, 14 out of 16 listed EPAs reached consensus as per pre-defined criteria (percentage agreement≥80%, median≥4, IQR≤1). The two EPAs that did not reach consensus were Platelet function disorders (PFDs): Diagnosing Qualitative Platelet disorders for resident PGY-1 and Von Willebrand Disease (VWD): Diagnosing Willebrand Disease for resident PGY-2, with each achieving an overall percentage of agreement of 75%. As for teaching strategies, desired consensus was readily achieved for strategies like demonstrations, supervised practice and reporting, case/slide-based learning, etc., but not achieved for lectures, ambulatory care in most EPAs. The assessment strategies that achieved the desired consensus mainly included case-based discussions (CBD), direct observations of procedural skill (DOPS), and slides preparation or comprehensive reporting, for many EPAs in this round. Eleven assessment strategies did not reach consensus in this round. There were no new suggestions for the items to be added except for a CBD and MCQs as assessment strategies for two of the EPAs.

Delphi Round 2

In round 2, all sixteen participants who participated in the first round were administered the modified online questionnaire, to seek consensus on the newly added item and on those items (EPAs, teaching, and assessment strategies) for which consensus was not achieved. The response rate was 81.25% as 13 out of 16 experts participated in the second round. The questionnaire also reported the achieved consensus from round 1 for each item. By the end of the second round, the EPAs that had failed to reach consensus still failed to meet the established criteria for acceptance. These two EPAs included Platelet function disorders (PFDs): Diagnosing Qualitative Platelet disorders for resident PGY-1 and Von Willebrand Disease (VWD): Diagnosing Willebrand Disease for resident PGY-2. Although both EPAs were considered ‘very important’ by most of the participants, the overall percentage of agreement for both had reached 62% and 77% only (Figure 2) compared to 75% for both in Round 1 of the Delphi process. In teaching strategies, lecturers still could not achieve a consensus for most of the EPAs, whereas 7 out of 11 assessment strategies from round 1 in various EPAs now attained consensus in the second round.

Figure 2. A 5-point Likert scale depicting the distribution of EPAs that did not reach > 80% consensus

The response stability of the two Delphi rounds was determined and it was found to be insignificant with a p-value of <0.05, as shown in Table 1.

Thus, the Delphi rounds were concluded with the acceptance and finalization of 14 EPAs and their respective teaching and assessment strategies, as shown in Table 2.

Discussion

This study illustrates systemic identification of essential EPAs and associated competencies in the context of laboratory hematology residency in Pakistan. In a 2024 study by Bryant, et al., Entrustable Professional Activities (EPAs) in Pathology residency contributed to clarifying performance expectations, offering more precise formative and summative feedback to residents, and improving program directors' insights into residents' strengths and areas for growth, thereby enhancing the overall feedback process and fostering better resident development ( 14 ). Sibtain, et al., conducted a study that evaluated a novel workplace-based assessment (WBA) model implemented in a postgraduate chemical pathology residency program at Aga Khan University, Pakistan, demonstrating its effectiveness in enhancing formative assessment, feedback quality, and trainee satisfaction. The findings suggested that this model can be adapted to other pathology specialties, including hematology, to support competency-based education and the development of Entrustable Professional Activities (EPAs) ( 15 ). In compliance with regulatory bodies personnel and testing requirements, Cotta CV, et al., did not report any adverse impact on test turnaround time of non-critical peripheral blood and body fluid reviews ( 16 ). In another study, the effect of EPAs on 3 transfusion reactions was evaluated ( 17 ).

During this study, seven faculty members from our institution actively participated in identifying and detailing EPAs. Among them, there were three full-time professors, two associate professors, and two assistant professors, with four individuals concurrently serving as supervisors of residency training programs or holding positions as program directors. Within the cohort of expert reviewers from 13 institutions nationwide, the average faculty work experience was 9.81 years with±6.5 standard deviation. Specifically, ten experts possessed an average of 4.94 years±7.2 standard deviation as residency program directors (PD). Notably, only two panelists had experience ranging from 2 to 5 years, while all other participants boasted teaching experience exceeding 5 years. Hence, it can be asserted with confidence that the initiative benefited from substantial teaching and assessment expertise right from the outset.

Additionally, expert reviewers from three out of four provinces in Pakistan were involved. Unfortunately, we faced challenges in Balochistan province as we couldn't find a suitable training institute with an expert reviewer having a minimum of 2 years of experience. Despite this limitation, our study remains a true representation of the country, given that only 5-7% of the Pakistani population resides in Balochistan ( 18 ).

At the conclusion of the second round, the EPAs that had previously struggled to achieve consensus still fell short of meeting the prescribed acceptance criteria. These two EPAs encompassed Platelet Function Disorders (PFDs): Diagnosing Qualitative Platelet Disorders for resident PGY-1 and Von Willebrand Disease (VWD): Diagnosing Willebrand Disease for resident PGY-2. Despite being deemed 'very important' by the majority of participants, the overall percentage agreement for both EPAs fell below the acceptable cut-off of 80%. This finding is particularly unexpected in the case of VWD, the prevalent inherited bleeding disorder. It narrowly missed meeting the acceptable criteria at 77%. It is conceivable that the criteria for acceptance were stringent (≥80%); perhaps, adding a couple of more expert reviewers might have pushed the results beyond the threshold for acceptance. Another possible explanation may be the absence of testing facilities for Von Willebrand Disease and Platelet Function Testing, which are not commonly available at institutions engaged in PG training in Pakistan. In contrast, a different specialized service, thrombophilia screening, successfully attained the targeted consensus. In 2019, the World Federation of Haemophilia (WFH) disclosed that 195,263 people were diagnosed with hemophilia in 115 countries. Pakistan accounted for over 2000 of these cases. However, this figure might not provide an accurate depiction of the true scenario, as several studies have highlighted a considerable occurrence of bleeding disorders in the country ( 19 , 20 ). Indeed, von Willebrand Disease (type 3) was found to be the most prevalent inherited autosomal recessive bleeding disorder, accounting for 34% of cases ( 19 ).

In teaching strategies, lectures completely failed to achieve consensus. Lectures are often deemed less useful in contemporary education for various reasons. The traditional lecture format, characterized by a one-sided flow of information, tends to foster passive learning rather than active engagement. With an emphasis on diverse learning styles, limited interaction, and short attention spans, lecturers may struggle to accommodate the needs and preferences of modern learners ( 21 ). Advancements in technology have also opened alternative avenues for accessing information, offering interactive and multimedia resources that can cater to a broader range of learning preferences. Additionally, there is a growing recognition of the importance of real-world application, critical thinking skills, and inclusive educational practices, areas where traditional lectures may fall short. As educational models evolve to embrace more student-centered approaches, lectures are being reconsidered in favor of methods that promote collaboration, practical application, and a deeper understanding of subject matter. Another teaching strategy that failed was simulation on a mannequin for bone marrow biopsies; again, authors attribute this failure to a lack of resources in most parts of the country.

Four assessment strategies could not achieve consensus at the conclusion of both Delphi rounds: 1) demonstration for the collection of blood products, 2) CBD for leukemias and preparation of comprehensive reports for 3) von Willebrand disease, and for 4) non-hematopoietic tumors. As far as preparation of comprehensive reports as assessment strategies are concerned, perhaps attending pathologists prefer obtaining a more profound understanding of the process novice learners experience when evaluating a case, providing an opportunity to teach a more tailored and efficient approach for each resident.

Our study aligns with ACGME's competency-based medical education principles by identifying core EPAs specific to hematology training in a low-resource setting. The integration of these EPAs into structured training not only enhances resident autonomy but also standardizes assessment strategies, enabling a more objective evaluation of readiness for independent practice. While "medical knowledge" appears to be overrepresented, it was intentionally included in most items, as the authors believe that without a strong foundation in medical knowledge, other competencies cannot be effectively achieved.

Limitations and future recommendations

Low response rate and dropout were a limitation of our study due to online Delphi. Since this was a single-center study, the results cannot be generalized. Multicentric or collaborative international studies regarding the development and implementation of EPAs in hematology in various contexts is recommended. An important limitation associated with EPAs is the potential challenge of standardizing assessments across various contexts and disciplines. Achieving a harmonious balance between context specificity and generalizability presents an intricate undertaking, necessitating additional research and collaborative efforts to tackle this issue. The implementation of EPAs may demand substantial resources, encompassing time, personnel, and technology. It is imperative for institutions and organizations to thoughtfully evaluate these resource implications to guarantee the long-term sustainability and efficacy of programs based on EPAs ( 22 ).

Conclusion

In conclusion, this study successfully tackled the recognized gap in knowledge by systematically identifying essential EPAs and their associated competencies, as well as delineating effective teaching and assessment strategies tailored to the context of hematology residency programs in Pakistan. Utilizing the modified Delphi method and engaging with the insights of experts across the country, we have established a comprehensive framework that aligns with the unique requirements of hematology training. The proposed EPAs contribute to the advancement of competency-based medical education in hematology. This framework provides valuable guidance for curriculum development, ensuring the cultivation of well-rounded and proficient hematologists in the Pakistani healthcare landscape. The collaborative effort and consensus achieved through this study pave the way for further enhancements in the training and evaluation of future hematology professionals.

Acknowledgements

We express our gratitude to Ms. Ayesha Sadiqa for statistical support.

Funding

Not applicable.

Authors’ contribution

M.SS and JR designed and planned the study, collected the data and wrote the initial draft of the manuscript. SB reviewed the manuscript. HH and BM contributed to the conception of the study and reviewed the manuscript critically. All authors reviewed the manuscript extensively and approved the final version.

Conflict of interests

The authors declare that they have no competing interests.

Declaration on the use of AI

The authors of this manuscript declare that no artificial intelligence (AI) was used during the writing process.

References

1. McCloskey CB, Domen RE, Conran RM, Hoffman RD, Post MD, Brissette MD, et al. Entrustable professional activities for pathology: recommendations from the College of American Pathologists Graduate Medical Education Committee. Acad Pathol. 2017; 4:2374289517714283.
2. Tu CY, Huang KM, Cheng CH, Lin WJ, Liu CH, Yang CW. Development, implementation, and evaluation of entrustable professional activities (EPAs) for medical radiation technologists in Taiwan: a nationwide experience. BMC Med Educ. 2024; 24(1):95.
3. Ten Cate O, Chen HC, Hoff RG, Peters H, Bok H, van der Schaaf M. Curriculum development for the workplace using entrustable professional activities (EPAs): AMEE guide no. 99. Med Teach. 2015; 37(11):983-1002.
4. Peters H, Holzhausen Y, Boscardin C, Ten Cate O, Chen HC. Twelve tips for the implementation of EPAs for assessment and entrustment decisions. Med Teach. 2017; 39(8):802-7.
5. Shorey S, Lau TC, Lau ST, Ang E. Entrustable professional activities in health care education: a scoping review. Med Educ. 2019; 53(8):766-77.
6. Alharbi LA, Cheikh M, Alotaibi ME, Alkhotani AA, Alim HM, Almalki F, et al. Developing and Validating Entrustable Professional Activities (EPAs) for Rheumatology Fellowship Training Programs in Saudi Arabia: A Delphi Study. Advances in Medical Education and Practice. 2024; 15:845-56.
7. White K, Qualtieri J, Courville EL, Beck RC, Alobeid B, Czuchlewski DR, et al. Entrustable professional activities in hematopathology pathology fellowship training: consensus design and proposal. Acad Pathol. 2021; 8:2374289521990823.
8. Ahmed A, Aman A, Khan S, Khan MF, Ullah H, Ahmed SSh, et al. Improving Postgraduate Anaesthesiology Training in a Low-Middle Income Country by Embracing the New Trends in Medical Education. Journal of the College of Physicians and Surgeons--Pakistan: JCPSP.  2023; 33(7):815-9.
9. Shaikh AJ, Ikram B. Enhancing Postgraduate Medical Education: Upscaling and Recognition of Pakistan’s Training. Journal of the College of Physicians and Surgeons Pakistan. 2024; 34(7):860-1.
10. Vaqar M, Tariq M, Khan MR, Khan S, Riaz Q, Mahmood S, et al. A Journey of Innovation: 40 years of Pioneering Medical Education at the the Aga Khan University Medical College in Karachi, Pakistan. Postgrad Med J. 2024; 100(1183):350-7.
11. Edgar L, McLean S, Hogan SO, Hamstra S, Holmboe ES. The milestones guidebook. USA: Accreditation Council for Graduate Medical Education; 2020.
12. Diamond IR, Grant RC, Feldman BM, Pencharz PB, Ling SC, Moore AM, et al. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. J Clin Epidemiol. 2014; 67(4):401-9.
13. Jünger S, Payne SA, Brine J, Radbruch L, Brearley SG. Guidance on Conducting and REporting DElphi Studies (CREDES) in palliative care: Recommendations based on a methodological systematic review. Palliat Med. 2017; 31(8):684-706.
14. Bryant BH, Anderson SR, Brissette M, Childs JM, Gratzinger D, Johnson K, et al. National pilot of entrustable professional activities in pathology residency training. Academic Pathology. 2024; 11(2):100110.
15. Sibtain A, Asad KFM, Hafsa M, Habib KA, Imran S, Farooq Gh, et al. From Practice to Proficiency: Evaluation of a Novel Workplace-Based Assessment (WBA) in a Postgraduate Chemical Pathology Residency Program in Pakistan. EJIFCC. 2024; 35(2):111-9.
16. Cotta CV, Ondrejka SL, Nakashima MO, Theil KS. Pathology Residents as Testing Personnel in the Hematology Laboratory: Developing Entrustable Professional Activities. Arch Pathol Lab Med. 2022; 146(7):894-902.
17. Ju JY, Wehrli G. The effect of entrustable professional activities on pathology resident confidence in blood banking/transfusion medicine. Transfusion. 2020; 60(5):912-7.
18. Ahmed M, Baloch A. Political Economy of Balochistan, Pakistan: A Critical Review. Eur Sci J. 2015; 11(14):1-20.
19. Naz A, Jamal MY, Amanat S, Din ujjan I, Najmuddin A, Patel H, et al. Autosomal recessive inherited bleeding disorders in Pakistan: a cross-sectional study from selected regions. Orphanet J Rare Dis. 2017; 12:1-7.
20. Hussain S, Baloch S, Parvin A, Najmuddin A, Musheer F, Junaid M, et al. Inherited Bleeding Disorders—Experience of a Not-for-Profit Organization in Pakistan. Clin Appl Thromb Hemost. 2018; 24(8):1241-8.
21. Bligh DA. What’s the Use of Lectures? Middlesex: Penguin Education; 1971.
22. El-Haddad C, Damodaran A, McNeil HP, Hu W. The ABCs of entrustable professional activities: an overview of &#039;entrustable professional activities&#039; in medical education.  Intern Med J. 2016; 46(9):1006-10.