Introduction

Over the past decade, the education sector has experienced a significant rise in the adoption of technologies to enhance teaching and learning practices, leading to unprecedented access to student data ( 1 - 2 ). This influx of educational data has fostered the emergence of research fields such as Learning Analytics (LA) and educational data mining, which are crucial for data-informed decision-making in contemporary education settings. LA is a subset of technology-enhanced learning (TEL) and is generally defined as the application of data science techniques to predict educational outcomes and generate actionable insights to support student learning ( 3 - 4 ). It incorporates educators' perspectives by providing insights that can inform teaching practices ( 5 - 6 ). Siemens ( 7 ) describes LA as encompassing the measurement, collection, analysis, and reporting of data related to learners and their environments, aiming to optimize learning experiences. This multidisciplinary field integrates knowledge from learning science, statistical analysis, computer science, and human-centered design ( 8 ). Despite its established role in educational research, systematic applications of LA are limited, often confined to small areas within educational programs ( 9 ). Medical education, in particular, presents a unique opportunity for LA to enhance student learning outcomes and quality assurance practices, thanks to its specialized knowledge requirements and rigorous accreditation processes. The dynamic nature of medical education necessitates collaborative efforts among various stakeholders to effectively plan, implement, and assess educational processes ( 10 ). LA can facilitate quality improvement in medical education by offering insights at personal, collective, curricular, and institutional levels. It supports competency-based medical education (CBME), emphasizing learner outcomes and competencies across various contexts ( 11 ). However, challenges remain, including the lack of standardized frameworks for LA in medical education, ethical concerns regarding data protection, technical complexities in data analysis, and the need for effective visualizations that aid interpretation ( 12 ). Addressing these challenges requires an interdisciplinary approach that considers the diverse perspectives of all stakeholders involved in medical education.

WHY MEDICAL EDUCATION?

Medical education aims to train future physicians and healthcare professionals, providing them with essential knowledge, skills, attitudes, and values necessary for disease prevention, health promotion, and the advancement of medical science ( 13 ). It encompasses various stages, including pre-medical preparation and undergraduate and postgraduate education ( 14 ). This field is dynamic, involving multiple stakeholders such as students, faculty, patients, and healthcare systems, being influenced by societal needs, technological advancements, resource availability, and ethical standards.

Medical education is a lifelong process that should adapt to rapid changes, including new diseases, therapies, and the growing demand for skilled healthcare professionals ( 13 ). Unlike other disciplines, medical education requires a rigorous integration of biological, psychological, social, and cultural factors related to health, alongside a strong ethical framework.

Students need to master a wide range of disciplines and increasingly utilize online tools to track performance and learning outcomes. However, effectively leveraging the vast data available in educational settings remains a challenge ( 15 ). Learning Analytics (LA) presents a promising solution to enhance medical education by utilizing digital data for evidence-based evaluations.

The diverse curricula across medical institutions complicate the assessment of teaching efficacy. LA can provide insights into different methodologies, aiding educators in making informed decisions about curriculum development ( 16 ). As curricula expand, tracking the students’ progress manually becomes burdensome, highlighting the need for LA to offer personalized learning experiences ( 17 , 18 ). By analyzing performance data, educators can identify students needing additional support and adjust teaching methods accordingly.

Additionally, the reliance on high-stakes summative assessments can influence genuine learning. LA can transform formative assessment practices, allowing for ongoing evaluation and feedback and promoting deeper engagement with the learning process ( 19 , 20 ). This shift emphasizes a holistic approach to medical education, focusing on improving learning outcomes and patient care.

WHY LA IN MEDICAL EDUCATION?

Learning Analytics (LA) offers significant potential in medical education by transforming how student learning is supported and integrating data-informed decision-making into teaching practices. LA empowers students to adapt to the fast-changing medical field by providing data-driven insights that personalize their learning experiences. Unlike traditional assessments, LA evaluates student interactions in simulations, case studies, and clinical scenarios, enabling instructors to enhance essential skills such as diagnostic reasoning, communication, and patient-centered care ( 21 - 23 ). This capability is crucial for managing complex decisions, effective collaboration, and ethical considerations in medical practice. Additionally, LA can identify students in need of extra support or stress management, thereby improving their well-being and performance.

Despite these advantages, there are gaps in understanding the current research landscape regarding LA in medical education and what constitutes effective implementation. A systematic review is proposed to address these gaps by synthesizing existing literature on LA in this field. The review aims to identify opportunities and challenges in applying LA, proposing strategies for successful integration into medical curricula. Ultimately, this research seeks to provide a comprehensive overview of current knowledge, highlight potential research pathways, and contribute to the advancement of LA in medical education.

Methods

This systematic review aims to provide a comprehensive overview of the current state of LA in medical education, exploring its applications, benefits, challenges, and future directions. Using Gough et al.’s ( 24 ) search strategy, several key stages were followed, including identifying the research question, developing the search strategy, establishing inclusion and exclusion criteria, selecting studies and quality assessment, extracting data, and descriptively synthesizing the findings.

Research question

How can learning analytics be effectively utilized to enhance educational practices and outcomes in medical education?

Search strategy

The systematic review was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyzes (PRISMA) guidelines. The following electronic databases were searched in June 2023 without any time restrictions on publication: ProQuest, Scopus, ERIC, Web of Science, PubMed, and Science Direct.

A search query was developed using keywords related to (a) learning analytics and (b) medical education. For learning analytics, the following terms were used: "learn* analytic*" OR "learn* analy?is" OR "teach* analytic*" OR "education* analytic*" OR "data mining" OR "big data". To address the medical education aspect, we included the terms "medical education" OR "medical training". For example, in the Scopus database the following search string was applied: ("learn* analytic*" OR "learn* analy?is" OR "teach* analytic*" OR "education* analytic*" OR "data mining" OR "big data") AND ("medical education" OR "medical training" OR " health profession education").

Inclusion and exclusion criteria

Studies were included in the screening if they explicitly incorporated the use of Learning Analytics (LA) within medical education settings. We imposed no restrictions based on the date of publication, location, or language, allowing for a comprehensive review of the literature. Only published journal articles and dissertations were considered, while editorials, commentaries, book chapters, and news articles were excluded. Additionally, studies that focused on medical education graduates were excluded to maintain our emphasis on LA during the active learning phase of students. This approach ensured that the included studies directly addressed both LA and medical education, thereby providing a targeted and relevant exploration of the research questions.

Study selection and Quality assessment

The results of the search query and selection process are illustrated in Figure 1. The database search query yielded a total of 1092 records. Additionally, three articles were identified through manual searching of Google Scholar, bringing the total number of references to 1095. Duplicate records (n=543) were then removed, leaving 552 titles and abstracts to be screened.

Figure 1. The PRISMA flow diagram of this study

After screening the titles and abstracts, 530 articles unrelated to LA and medical education were excluded. The full texts of the remaining 22 articles were assessed for eligibility. Following the application of the inclusion/exclusion criteria, 10 articles were excluded for reasons such as editorials/commentaries, lack of focus on LA, or lack of relation to medical education.

In total, 12 studies were selected for inclusion in the synthesis. The PRISMA flow diagram (Figure 1) depicts the article identification, screening, eligibility, and inclusion processes.

The studies included in this systematic review were evaluated using a range of standardized checklists, chosen according to the study design. For observational and exploratory descriptive studies, the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) checklist was applied, allowing for a comprehensive assessment of the context, sampling techniques, and the clarity of the reported findings in the studies ( 25 ). In the case of the mixed methods study, the MMAT (Mixed Methods Appraisal Tool) was employed to verify the rigor of the design and proper integration of both qualitative and quantitative components ( 26 ). Retrospective cohort studies were assessed using the CASP (Critical Appraisal Skills Program) checklist for cohort studies, which focused on identifying and managing confounding variables, as well as the reliability of data collection and analysis ( 27 ).

To ensure thoroughness and consistency, two independent reviewers (the first and second authors) conducted separate assessments of each study. Discrepancies were discussed and resolved, with additional input from a research methodology expert when needed. This comprehensive evaluation process ensured that the quality assessments were precise and aligned with the specific methodological approaches of the studies.

Data extraction

Data were collected from the included studies using standardized forms in Microsoft Excel. The information is shown in Table 1, describing the basic characteristics of each study, such as the authors, year of publication, country conducted, number of participants, medical discipline studied, platform, duration of participation, data extraction tool, type of data, and goal of data.

Descriptive synthesis of findings

The data from the 12 selected articles were analyzed with the findings described in detail in the Results section of the article.

Ethical consideration

In conducting this systematic review, ethical considerations were paramount to ensure the integrity and credibility of the research process. Since this review involved the analysis of existing literature rather than direct intervention with human participants, we adhered to ethical guidelines for the responsible use of published data. All studies included were selected based on rigorous criteria, ensuring that they were complied with ethical standards in their original research. Additionally, potential conflicts of interest were disclosed, and the review process was designed to be transparent and reproducible.

Results

Search results

Following an extensive search across databases, a total of 1,095 studies were initially identified. Subsequently, 543 studies were removed as duplicates, after which the titles and abstracts of the remaining 552 studies were screened. Of these, 530 studies were excluded due to irrelevancy, leading to a detailed review of the full texts of 22 studies. After a thorough evaluation, 12 studies were deemed pertinent and included in the final descriptive analysis. This rigorous selection process ensures that the included studies are highly relevant to the systematic review of LA in medical education, reflecting a focused and comprehensive exploration of the available literature on the subject.

Description of the included studies

Table 1 presents the characteristics of the studies included in the study. Of the included studies, most were conducted in the USA ( 28 - 33 ), two in Turkey ( 34 , 35 ), and one each in Australia ( 36 ), Saudi Arabia ( 19 ), Germany ( 37 ), and Italy ( 38 ). The participants of the studies included in the systematic review were mainly medical students ( 19 , 28 , 35 - 38 ), followed by internal medicine residents ( 31 ), anesthesia residents ( 33 ), pharmacy students ( 34 ), and health professionals with different educational levels ( 29 ). Two studies did not report the participants’ characteristics ( 30 , 32 ). This investigation included 12 studies published between 2016 and 2023. The most recent study was published in 2023 ( 29 ), while the oldest one was published in 2016 ( 32 ). Most of the studies were published in 2020 and 2022, indicating the growing interest in LA in health education. The studies used different sampling methods to select the participants, such as convenience sampling ( 19 , 29 - 31 , 33 , 34 , 37 , 38 ), census sampling ( 28 ), and volunteer sampling ( 35 ). Two studies did not report their sampling methods ( 32 , 36 ). The sample size varied from 26 to 2806 participants, with a median of 342. Four studies did not mention their sample size ( 28 , 30 , 32 , 36 ). The studies also used different platforms to collect and analyze the data for LA. Three studies did not specify the platforms they used, while the others reported the use of various platforms, such as a custom-built software application ( 38 ), an online data platform ( 31 ), Aquifer online learning platform (formerly MedU) ( 29 ), computer program VQuest ( 37 ), google trends web application ( 32 ), and LMS ( 19 , 34 - 36 ). The studies varied in their research designs for investigating LA in health education. The most frequent designs were exploratory descriptive studies ( 35 , 38 ), observational descriptive studies ( 19 , 32 ), and mixed methods studies ( 36 , 37 ). The other designs were longitudinal descriptive study ( 31 ), correlational descriptive study ( 29 ), predictive modeling study ( 34 ), survey-based study ( 28 ), and retrospective cohort study ( 33 ). One study did not report its design clearly ( 30 ). The duration of participation varied according to the design and scope of each study, ranging between five weeks ( 34 ) to 6 years ( 32 ). Data demonstration formats were tabular data ( 19 , 29 , 31 , 32 , 34 , 36 - 38 ), chart data ( 29 , 38 ), graphic data ( 32 , 33 , 35 , 36 , 38 ), and figure data ( 34 ). Table 1 presents the characteristics of the included studies.

Interventions of the included studies

LA in medical education can be used to measure various aspects of students’ learning outcomes and behaviors. They included the students’ performance on the clinical case simulations and their perceptions of the virtual patients ( 38 ), the level of student engagement with virtual patient simulations ( 28 ), the resident performance on the reportable internal medicine sub-competencies ( 31 ), accuracy on the case multiple choice question ( 29 ), cognitive engagement ( 37 ), the frequency and volume of online searches related to specific topic ( 32 ), the learning habits of students ( 36 ), a measure of the students’ academic achievement and performance ( 19 ), students’ interaction with online materials ( 35 ), the audience retention ( 30 ), and milestone level ratings ( 33).

We analyzed and collected five types of data in this review: log, text, point, time, and user information. Log data were used in various ways, such as the number of clicks ( 29 ), simulation execution ( 28 ), forum activity ( 35 ), affected user ( 34 ), type of activity, source of activity ( 28 ), logins ( 29 , 34 , 35 , 37 , 38 ), search terms and Google Trends data ( 29 ), video view count ( 30 ) as well as student use of locally produced, optional, self-directed learning resources ( 36 ). Text data were used in writing summary statements per case based on the findings in the VP encounter ( 28 ), the correctness of answers, and class-wide face-to-face discussions ( 37 ). MCQ ( 28 ), quiz scores ( 31 ), scoring summary statement per case based on the findings in the VP encounter ( 28 ), formative assessment ( 36 ), performance assessment scores ( 31 ), and Milestone ratings ( 33 ) are related to point data ( 34 ). Time data refers to time spent on each page and task ( 35 ), percentage of time spent on each video ( 33 ), answering questions, assignments ( 37 ), and total time ( 35 ). User information data mentions username, affected user, event context, component, event name, description, origin ( 34 ), speaker allocation, speech act, and elicitation-response patterns ( 37 ). The characteristics of the LA data extracted in this review are reported in Table 1.

Tables 2 and 3 present a succinct summary of the varied goals and corresponding types of data employed in LA studies. Notably, the most prevalent goal was engagement, assessed through six distinct data types: activity, point, search data, time, text, and view count. Following closely, the identification of at-risk students involved the utilization of five types of data: point, view count, forum activity, time, and activity. In contrast, feedback to the teacher and learning path emerged as fewer common goals, each measured by two types of data: time and view count for feedback to the teacher, and activity and point for the learning path. The most common type of data was logging data, which was used to measure engagement, learning path, and at-risk students, while the least common type of data was text data, which was only used to measure engagement. The tables also present instances of overlap, underscoring the interconnectivity of data types across different goals in LA studies.

Discussion

In this systematic review, we explored the multifaceted landscape of LA in medical education, aiming to address its applications, benefits, challenges, and future trajectories. The discussion will unfold across distinct dimensions, including applications of LA, the nature of utilized data, perceived advantages and challenges, inherent limitations in existing research, and recommendations for the optimal use of LA in the context of medical education.

Applications of LA in Medical Education

The findings presented in Table 2 indicate that the primary role of LA in medical education lies in the evaluation of student engagement levels and identification of at-risk students. Student engagement emerges as a pivotal factor influencing both learning outcomes and overall student satisfaction ( 39 ). In the context of e-learning environments, engaged students actively participate in course discussions, are motivated to review and complete educational content, and demonstrate a proactive interest in the learning process ( 40 ). The results indicate that engagement, a multifaceted construct, can be effectively measured through six distinct types of activity data: scores, search queries, time spent, textual interactions, and the frequency of visits. This diverse array of indicators underscores the complexity of engagement and the necessity to capture it through various dimensions of students' behaviors in online learning environments. The collection and analysis of data on student engagement serve several crucial purposes. Firstly, it offers valuable guidance to both students and educators on strategies to enhance engagement and subsequently improve learning outcomes, thereby fostering a continuous feedback loop ( 41 ). Furthermore, by rendering students' involvement visible and tangible, the analysis of engagement data contributes to the elevation of students' motivation, self-regulation, and metacognitive skills ( 19 ). Lastly, it serves as a vital tool for the identification and support of students exhibiting low engagement, suboptimal performance, or those at risk of dropout ( 17 ).

Within medical education, the reasons for students to be placed or positioned as at-risk are frequent challenges such as financial constraints, family issues, or academic struggles. LA can be employed to set training support interventions to help mitigate student attrition more proactively. As the results of the systematic review demonstrate, analyzing parameters such as grades, observation frequency, participation in discussions, and the time spent on educational activities enables the early recognition of at-risk students. LA, thus, emerges as a pivotal tool for educators to promptly address learning difficulties, offering tailored interventions to enhance academic success. Furthermore, LA fosters self-awareness among at-risk students, allowing them to identify and work on their strengths and weaknesses in various learning domains. In the broader research landscape, continuous assessment ( 42 ), performance-based assessment ( 43 ), and regular feedback ( 41 ) stand out as recommended methods. By attending to learning fluency concerning at-risk students, LA not only improves academic results but also mitigates the risk of dropout, contributing to a more inclusive and supportive medical education environment.

Building upon the identification and support of at-risk students, the findings of this study underscore four additional pivotal goals achievable through LA in medical education, including enhancing participation, providing feedback, tracking learning habits, and identifying learning paths. Aligned with the study by Chen et al., the results affirm that LA facilitates the monitoring of student performance and behavior, enabling educators to discern those requiring additional support or intervention ( 44 ). Mortazavizadeh's research echoes this sentiment, emphasizing the role of LA in aiding teachers to identify the students' needs, strengths, and weaknesses, as well as offering tailored feedback and guidance ( 45 ). Banihashem further contributes to the discourse by recommending key components to teachers utilizing LA, encompassing feedback, self-regulation, motivation, monitoring, and assessment ( 46 ). The overarching implication is that LA serves as a valuable tool for teachers and educators in medical education, boosting their capacity to teach more effectively and preparing students for success in the medical profession.

In addition to teachers and educators, medical students can leverage LA to enhance their learning experience, despite not having direct control over LA systems. By reviewing performance data and analytics, assessment scores, and feedback, they gain valuable insights into their academic standing, identifying strengths and areas for improvement. Analyzing study habits and time allocation to learning activities empowers them to manage their time effectively. The use of LA enables medical students to pinpoint weaknesses, set personalized learning goals, and systematically track their progress. Adaptive learning platforms, tailoring content and assessments based on individual performance, further enhance their educational journey. Reflecting on study strategies and engaging with educational resources guided by analytics data optimizes their learning routine. Exam readiness insights contribute to informed preparation, and career exploration aligns seamlessly with identified strengths and interests. LA serves as a guiding force in their interaction with various resources, and in the face of challenges, seeking support from teachers or peers becomes an informed process. Ultimately, the integration of LA with self-awareness significantly enriches the overall academic experience for medical students.

Data in LA

a. Examination of Single and Multiple Sources of Data

The study findings highlight the diverse approaches in utilizing data sources for the evaluation of different criteria. Notably, the evaluation of each criterion can be undertaken through various data sources, demonstrating the versatility of data in analyzing learning. Some studies opt for a singular data source to assess learning. For instance, Alper Bayazit et al. ( 34 ) employed scores exclusively to identify at-risk students, while others focused solely on grades ( 31 , 33 ) or activity ( 38 ) to delineate the students' learning paths. Contrastingly, several studies embraced a more comprehensive approach by integrating two or more data sources. Saqr et al. ( 19 ), for instance, employed a combination of activity data, grades, time spent, and the number of visits to identify at-risk students, demonstrating the comprehension that can be achieved by combining types of data. Similarly, Leng and Pawelka ( 37 ) utilized activity, time, textual interactions, and the number of views data to measure engagement. Berman and Artino's study ( 28 ) incorporated activity, grades, and time data to measure participation levels. These instances highlight the fact that a one-size-fits-all approach for data types per criterion is not feasible. Instead, the augmentation of data sources not only enriches the assessment process but also enhances the depth of understanding across various educational criteria.

b. Data Overlapping

The reviewed studies exhibit both overlapping and variations in terms of data sources, highlighting the adaptability of certain data types across multiple criteria. In the analyzed literature, data related to grades and the number of observations ( 19 , 34 ), activity in discussion forums ( 19 ), and time ( 19 , 34 , 35 ) were commonly employed to identify at-risk students. Exploring students' involvement revealed an array of overlapping data sources, including activity data ( 19 , 37 ), grades ( 28 ), search data ( 29 ), time, and the number of observations ( 28 , 29 , 37 ), as well as text analysis ( 37 ). Furthermore, data originating from students’ activity serves varied purposes such as evaluating the level of participation ( 28 , 37 ), assessing the learning path ( 38 ), and identifying at-risk students ( 19 ). Similarly, the number of observations emerged as a multifunctional data source, utilized for providing feedback to teachers ( 30 ), identifying learning habits ( 36 ), pinpointing at-risk students ( 19 ), and measuring the level of student participation ( 19 , 28 , 29 ). These findings underscore the notion that certain data sources possess the flexibility to serve multiple purposes, offering a holistic perspective across diverse educational criteria, while others may be more specialized and unique in their application. The nuanced utilization of overlapping data sources underscores the complexity and depth of analysis afforded by LA in the context of medical education.

c. Systematic Continuous Data

A key strength of LA lies in its capacity to systematically access and analyze diverse and continuous data, as evidenced by the studies included in this systematic review. The temporal scope of data collection within these studies ranged from 5 weeks ( 34 ) to 6 years ( 32 ), demonstrating the adaptability of LA to different timeframes. This continuity in data acquisition empowers LA to discern and interpret meaningful patterns, trends, and relationships within the learning process. The ability to analyze data over extended periods facilitates the identification of long-term educational trajectories and provides a more comprehensive understanding of learning dynamics. Through this systematic and continuous approach, LA emerges as a potent tool capable of offering deeper and more reliable insights to enhance the overall learning experience, teaching methodologies, and the learning environment ( 46 ). The evolving landscape of medical education, characterized by a growing emphasis on continuous assessment, necessitates methods capable of professionally evaluating the learners' competencies across intricate domains such as clinical reasoning, communication skills, physical examination, and professionalism. The integration of LA within the framework of stealth assessment represents an innovative approach, enabling a nuanced and realistic analysis of skills and performance by examining any data generated by a medical student during various activities. By embracing systematic continuous data, LA contributes to a more dynamic and adaptive educational paradigm, aligning with the evolving needs of medical education.

d. Data Visualization

An additional dimension of LA in medical education involves the application of data visualization. Across the conducted studies, data analysis was visually represented in various forms, including tables (eight studies), charts (two studies), graphs (five studies), and figures (one study). The diversity in data presentation formats tries to cater for different audiences and contexts, enhancing the accessibility and relevance of the conveyed information. The choice of visualization methods is strategic, with tables being effective for displaying detailed numerical comparisons and relationships between experiments. Graphs prove valuable in illustrating trends, patterns, and data distributions, while figures excel in visualizing conceptual models. The combination of these visualization techniques proves advantageous for integrating multiple data sources and offering diverse perspectives as well as levels of analysis. In the realm of learning in medical education, data visualization serves as a potent tool for conveying accurate information in a more comprehensible manner, unveiling simple patterns and processes that can be discerned and understood independently, ultimately fostering improved comprehension and facilitating imaginative engagement.

Opportunities and Challenges of LA in Medical Education

The integration of LA in medical education presents numerous advantages, including enhancing decision-making processes and improving learning outcomes ( 38 ). It facilitates personalized and adaptive learning, tailoring educational experiences to individual needs and pacing, thereby optimizing learning outcomes ( 31 ). LA also proves instrumental in streamlining evaluation and feedback processes ( 29 ). Wise emphasizes the significance of data-driven decision-making in refining the teaching-learning process, as data extracted from the learning journey provide valuable insights for informed decision-making ( 47 ). Dawson et al. highlight LA's role in supporting formative and summative assessments, offering feedback to learners and instructors, and guiding decisions about curriculum design and delivery ( 48 ). Li and Wong underscore the student-centered approach of personalized learning, addressing issues of student engagement and tailoring experiences to individual needs, plus fostering adaptive content and interactive learning ( 49 ).

However, the benefits of LA in medical education are accompanied by challenges and potential disadvantages. Foremost among them are ethical and privacy concerns arising from the collection, storage, analysis, as well as the use of sensitive and personal student data ( 29 , 31 ). Slade and Prinsloo delved into these ethical issues, noting that privacy concerns extend beyond data management to encompass the ethical implications of tracking and analyzing sensitive information about patients and medical cases ( 50 ). Also, the overreliance on quantitative criteria may oversimplify the complex nature of clinical competencies and professional judgment, potentially de-emphasizing critical thinking and clinical reasoning skills. Resistance from faculty and students to adopt new technological tools poses another challenge, hindering the seamless integration of LA into existing medical education practices. Moreover, the rapid pace of technological advancements may lead to obsolescence, requiring continuous updates and training, creating logistical and financial challenges for institutions. Striking a delicate balance between leveraging the benefits of LA and addressing potential drawbacks is crucial for the effective and ethical implementation of this technology in medical education.

Limitations Identified in the Reviewed Studies

Several limitations were identified across the reviewed studies, influencing the generalizability, timing, performance, measurement tools, data quality, and outcome evaluation. Some studies faced challenges in generalizability due to small sample sizes, specific focus on certain fields or tools, or use of a limited set of evaluation tools ( 31 , 38 ). Another limitation involved the timing of LA performance, where the identification of at-risk students occurred immediately before the end-of-semester exam, leading to delayed identification ( 34 ). Akifeh et al. faced limitations in data quality by utilizing google trends, which may lack real features, contain unpublished algorithms, and rely on biased samples, potentially impacting data relevance, completeness, and representativeness. Additionally, some studies lacked outcome data to comprehensively evaluate the impact of the LA system on the learning and performance of learners ( 31 ). These identified limitations underscore the importance of addressing methodological constraints in future research endeavors, promoting robust study designs, diverse samples, and comprehensive outcome assessments to enhance the reliability and applicability of findings in the realm of LA in medical education.

Since completing our literature search, the researchers have made a concerted effort to stay current in this rapidly expanding field through continuous reading, engaging in discussions with field experts, and subscribing to publication alerts from databases. Nevertheless, due to the dynamic nature of this area of study, there may be studies that have been published or are currently undergoing submission that were not covered in this review. Additionally, despite our efforts to encompass all relevant articles by employing a wide range of carefully selected search terms, it is possible that some pertinent studies may have inadvertently been omitted.

Suggestions for Effective Integration of LA in Medical Education

Based on the comprehensive review and analysis of the included studies, the following recommendations are posed to enhance the efficiency of LA in medical education:

o Larger and More Diverse Samples: Employing larger and more diverse samples of students, teachers, and fields to enhance the generalizability and validity of study results ( 31 , 32 , 38).

o Ethical and Practical Considerations: Paying meticulous attention to the ethical and practical considerations associated with LA data usage, addressing issues such as privacy, consent, ownership, access, interpretation, and feedback provision ( 29 , 37 , 38).

o Innovative Learning Assessment Tools: Designing, developing, and testing new or improved learning assessment tools that support various learning scenarios, including visual, multimodal, or interaction-based assessment methods ( 19 , 28 , 29 , 35 , 36).

o Comprehensive Data Analysis: Providing a more detailed, specific, and relevant examination of data, integrating insights from other tools such as Click Stream and google trends to augment the depth of analysis ( 29 , 32).

o Structured and Timely Assessment: Leveraging LA to deliver more structured, timely, and meaningful assessment as well as feedback to both learners and teachers, focusing on the identification and support of at-risk students ( 34 - 36).

o Measurement of Participation and Motivation: Utilizing LA to measure, strengthen, and assess students' participation and motivation in the learning process, fostering a more engaged and dynamic educational environment ( 19 , 28 , 31).

o Professional Development for Educators: Implementing ongoing training and professional development programs for educators to familiarize them with LA tools and methodologies, ensuring they can effectively integrate these technologies into their teaching practices ( 19 , 31).

o Collaboration Across Disciplines: Encouraging interdisciplinary collaboration among faculty members to share best practices and insights related to LA, fostering a more holistic approach to its implementation in medical education ( 36 - 38).

Feedback Loops for Continuous Improvement: Establishing feedback loops that allow students and educators to provide input on the effectiveness of LA tools and strategies, enabling continuous refinement and adaptation of these technologies to better meet educational needs ( 29 , 32).

These suggestions collectively contribute to the ongoing optimization of LA in medical education, emphasizing ethical practices, methodological robustness, and practical applicability to enhance learning outcomes.

Conclusion

This systematic review underscored the multifaceted perspective of LA in medical education, unveiling both its potential benefits and formidable challenges. The positive aspects of LA in medical education capture its capacity to personalize learning experiences, proactively identify at-risk students for timely interventions, and contribute to evidence-based decision-making for continuous curriculum improvement. Particularly in the realm of measuring student engagement, LA not only offers practical insights for immediate educational enhancements but also aligns with broader goals, including the promotion of student motivation, self-directed learning, and the implementation of early intervention strategies to ensure the sustained progress of education. However, amidst these promising prospects, critical concerns demand attention. Privacy issues, the potential risk of oversimplifying clinical competencies, and likelihood of resistance to technology adoption are significant challenges that cannot be overlooked. This synthesis of literature underscores the crucial importance of striking the right balance between data-driven insights and the human elements inherent in medical education to successfully integrate LA.

Looking ahead, it is imperative for educators, policymakers, and technologists to collaboratively explore these complexities and cultivate a comprehensive understanding of the role of LA in shaping the future of medical education. This exploration should be undertaken with mindfulness of ethical and legal implications, recognizing the profound effect of LA on educational methods. By navigating these challenges thoughtfully, the integration of LA has the potential to revolutionize medical education, fostering an environment that seamlessly merges technological advancements with the core values of effective, ethical, and human-centric education.

Authors’ contributions

MK, NK, MA: designed the study and supervised the manuscript, and FS: drafted and corrected the manuscript. ET and MK conceived the study and are responsible for all results. SS performed the search, in consultation with MK and ET. ET, SD, SS, and MK wrote the first draft and critical revision of the manuscript. All authors read and approved the final manuscript.

Conflicts of Interest

The authors declare no conflicts of interest.

References

1. Liu Q, Geertshuis S, Grainger R. Understanding academics' adoption of learning technologies: A systematic review. Computers & Education. 2020; 151:103857.
2. Ang KLM, Ge FL, Seng KP. Big educational data & analytics: Survey, architecture and challenges. IEEE access. 2020; 8:116392-414.
3. Ferguson R. Ethical Challenges for Learning Analytics. Journal of Learning Analytics. 2019; 6(3):25-30.
4. Junco R, Clem C. Predicting course outcomes with digital textbook usage data. The Internet and Higher Education. 2015; 27:54-63.
5. Xing W, Guo R, Petakovic E, Goggins S. Participation-based student final performance prediction model through interpretable Genetic Programming: Integrating learning analytics, educational data mining and theory. Computers in human behavior. 2015; 47:168-81.
6. Drachsler H, Kalz M. The MOOC and learning analytics innovation cycle (MOLAC): A reflective summary of ongoing research and its challenges. Journal of Computer Assisted Learning. 2016; 32(3):281-90.
7. Rubel A, Jones KM. Student privacy in learning analytics: An information ethics perspective. The information society. 2016; 32(2):143-59.
8. Siemens G. Learning analytics and open, flexible, and distance learning. Distance Education. 2019; 40(3):414-8.
9. Mayer RE, Alexander PA. Handbook of research on learning and instruction. USA: Taylor & Francis; 2016.
10. USA; 2019.
11. Cullen M, Cadogan C, George S, Murphy S, Freeney S, Fitzpatrick R, et al. Key stakeholders’ views, experiences and expectations of patient and public involvement in healthcare professions’ education: a qualitative study. BMC Medical Education. 2022; 22(1):305.
12. Chan T, Sebok‐Syer S, Thoma B, Wise A, Sherbino J, Pusic M. Learning analytics in medical education assessment: the past, the present, and the future. AEM Education and Training. 2018; 2(2):178-87.
13. Buja LM. Medical education today: all that glitters is not gold. BMC Medical Education. 2019; 19(1):1-11.
14. Louw BC. Educating Physicians for the 21 st Century: Learning from the Experiences of ‘Systems Citizens’. USA: The Pennsylvania State University; 2020.
15. Cheung SK, Kwok LF, Phusavat K, Yang HH. Shaping the future learning environments with smart elements: challenges and opportunities. International Journal of Educational Technology in Higher Education. 2021; 18:1-9.
16. Holmboe ES, Yamazaki K, Nasca TJ, Hamstra SJ. Using longitudinal milestones data and learning analytics to facilitate the professional development of residents: early lessons from three specialties. Acad Med. 2020; 95(1):97.
17. Klašnja-Milićević A, Ivanović M, Stantić B. Designing personalized learning environments—The role of learning analytics. Vietnam Journal of Computer Science. 2020; 7(3):231-50.
18. Corral J, Konstantinidis ST, Bamidis PD. Learning analytics, education data mining, and personalization in health professions education. Digital Innovations in Healthcare Education and Training. USA: Elsevier; 2021.
19. Saqr M, Fors U, Tedre M. How learning analytics can early predict under-achieving students in a blended medical education course. Med Teach. 2017; 39(7):757-67.
20. Lee AS, Donoff C, Ross S. Using Learning Analytics to Examine Differences in Assessment Forms from Continuous Versus Episodic Supervisors of Family Medicine Residents. Journal of Graduate Medical Education. 2022; 14(5):606-12.
21. Ellaway RH, Topps D, Pusic M. Data, big and small: emerging challenges to medical education scholarship. Acad Med. 2019; 94(1):31-6.
22. Almansour M, Almoayad F. Exploring challenges and perceptions in the learning environment: an online qualitative study of medical students. BMC Medical Education. 2024; 24(1):147.
23. Delungahawatta T, Dunne SS, Hyde S, Halpenny L, McGrath D, O’Regan A, et al. Advances in e-learning in undergraduate clinical medicine: a systematic review. BMC Medical Education. 2022; 22(1):711.
24. Gough D, Thomas J, Oliver S. An introduction to systematic reviews. USA: SAGE; 2017.
25. Skrivankova VW, Richmond RC, Woolf BA, Yarmolinsky J, Davies NM, Swanson SA, et al. Strengthening the reporting of observational studies in epidemiology using Mendelian randomization: the STROBE-MR statement. JAMA. 2021; 326(16):1614-21.
26. Oliveira JL, Magalhães AM, Matsuda LM, Santos JL, Souto RQ, Riboldi CD, et al. Mixed methods appraisal tool: Strengthening the methodological rigor of mixed methods research studies in nursing. Texto & Contexto-Enfermagem. 2021; 30:e20200603.
27. Long HA, French DP, Brooks JM. Optimising the value of the critical appraisal skills programme (CASP) tool for quality appraisal in qualitative evidence synthesis. Research Methods in Medicine & Health Sciences. 2020; 1(1):31-42.
28. Berman NB, Artino AR. Development and initial validation of an online engagement metric using virtual patients. BMC Medical Education. 2018; 18(1):1-8.
29. Cirigliano MM, Guthrie CD, Pusic MV. Click-level learning analytics in an online medical education learning platform. Teaching and Learning in Medicine. 2020; 32(4):410-21.
30. Lau KV, Farooque P, Leydon G, Schwartz ML, Sadler RM, Moeller JJ. Using learning analytics to evaluate a video-based lecture series. Med Teach. 2018; 40(1):91-8.
31. Park YS, Zar F, Tekian A. Synthesizing and reporting milestones-based learner analytics: validity evidence from a longitudinal cohort of internal medicine residents. Acad Med. 2020; 95(4):599-608.
32. Phelan N, Davy S, O'Keeffe GW, Barry DS. Googling in anatomy education: Can google trends inform educators of national online search patterns of anatomical syllabi? Anatomical Sciences Education. 2017; 10(2):152-9.
33. Tanaka P, Park YS, Roby J, Ahn K, Kakazu C, Udani A, et al. Milestone learning trajectories of residents at five anesthesiology residency programs. Teaching and Learning in Medicine. 2021; 33(3):304-13.
34. Bayazit A, Apaydin N, Gonullu I. Predicting at-risk students in an online flipped anatomy course using learning analytics. Education Sciences. 2022; 12(9):581.
35. Bayazit A, Ilgaz H, Gönüllü İ, Erden Ş. Profiling students via clustering in a flipped clinical skills course using learning analytics. Med Teach. 2023; 45(7):724-31.
36. Scott K, Morris A, Marais B. Medical student use of digital learning resources. The clinical teacher. 2018; 15(1):29-33.
37. de Leng B, Pawelka F. The use of learning dashboards to support complex in-class pedagogical scenarios in medical training: how do they influence students’ cognitive engagement? Research and Practice in Technology Enhanced Learning. 2020; 15(1):1-17.
38. Furlan R, Gatti M, Mene R, Shiffer D, Marchiori C, Giaj Levra A, et al. Learning Analytics Applied to Clinical Diagnostic Reasoning Using a Natural Language Processing–Based Virtual Patient Simulator: Case Study. JMIR Medical Education. 2022; 8(1):e24372.
39. Nortvig AM, Petersen AK, Balle SH. A literature review of the factors influencing e‑learning and blended learning in relation to learning outcome, student satisfaction and engagement. Electronic Journal of E-learning. 2018; 16(1):46-55.
40. Rajabalee BY, Santally MI, Rennie F. A study of the relationship between students’ engagement and their academic performances in an eLearning environment. E-learning and Digital Media. 2020; 17(1):1-20.
41. Corrin L, de Barba P. Exploring students’ interpretation of feedback delivered through learning analytics dashboards. In Hegarty J, McDonald S, Loke K. (Eds.). Rhetoric and Reality: Critical perspectives on educational technology. Proceedings ascilite, Dunedin; 2014. pp. 629-33. Available from: https://melbourne-cshe unimelb edu au/research/research-projects/edutech/learning-analytics-dashboards
42. Aftab MT, Tariq MH. Continuous assessment as a good motivational tool in medical education. Acta Med Acad. 2018; 47(1):76-81.
43. Williams P. AI, Analytics and a New Assessment Model for Universities. Education Sciences. 2023; 13(10):1040.
44. Chen Y, Zheng Q, Ji S, Tian F, Zhu H, Liu M. Identifying at-risk students based on the phased prediction model. Knowledge and Information Systems. 2020; 62:987-1003.
45. Movahedzadeh SH, Hassani M. Analyze novice teachers' experiences of teaching challenges in multigrade classrooms. New Educational Approaches. 2021; 16(1):115-40.
46. Banihashem S. Identifying components of learning analytics in education and providing a conceptual framework for optimizing learning. Technology of Education Journal (TEJ).. 2020; 14(4):937-48.
47. Wise AF. Learning analytics: Using data-informed decision-making to improve teaching and learning: Maximizing student engagement, motivation, and learning. USA: Contemporary Technologies in Education; 2019.
48. USA; 2014.
49. Thailand: Springer; 2020.
50. Slade S, Prinsloo P. Learning analytics: Ethical issues and dilemmas. American Behavioral Scientist. 2013; 57(10):1510-29.