Introduction

Evidence-based medicine (EBM) emerged as a movement in response to limitations in comprehension and application of published evidence. EBM's early focus was on training physicians to evaluate and apply published research to improve clinical treatment ( 1 ).

EBM has emphasized the importance of combining critical appraisal of the evidence with the principles and preferences of the patient through shared decision-making ( 1 ). Critical appraisal is one of the essential components of EBM, including a hands-on course for thorough and regular investigation of research to determine its validity, value, and applicability in assisting clinicians in making critical clinical decisions ( 2 ).

To deal with both the information explosion and fundamental human brain limits in processing information, applying the EBM principle of critical appraisal to identify high-quality research on behalf of doctors is required ( 1 ). Critical appraisal is an essential skill for medical students as future physicians to improve their diagnosis, patient care, and decision-making ability ( 3 ).

Undergraduate medical students are mandated to own skills in critically appraising scientific literature to make precise decisions for their patients. Critical appraisal skills training is necessary for taking care of patients effectually and efficiently and must be a part of every medical education curriculum ( 4 , 5 ). It is an effective learning method for students that can meet the changing needs of the treatment and education system ( 6 ). A more recent survey suggested that the undergraduate medical education curriculum should join a structured course plan in the critical appraisal field ( 7 ).

Even though the EBM principle of critical appraisal training is now required in all physician training programs, there is still no harmony on how to ensure best that these skills are appropriately taught for lifelong learning. The most effective EBM principle of critical appraisal teaching methods is unknown, and there is limited evidence that specific teaching strategies contribute to better patient outcomes ( 8 ). Some medical school curricula are limited in their ability to deliver comprehensive EBM training due to a lack of teachers and accompanying materials ( 9 ).

The importance of instruction in critical appraisal, the essential field of appraisal teaching to undergraduate medical students, is not defined well. In spite of some literature on teaching courses and broader critical appraisal of scientific literature, the uptake of knowledge emerging from this process is not well known. In traditional EBM training, most modules develop answerable questions and search available literature. Developing critical appraisal skills has been a portion of the training in limited sets, and medical students usually learn this skill through informal settings and hidden curricula ( 10 , 11 ). Additionally, Tabari, et al. demonstrated that educational interventions considerably impacted students' knowledge and attitudes ( 12 ).

This article presents an innovative model to meet the needs of assessment, initiation, and development of a multi-media framework and online workshop in critical appraisal teaching to better prepare the medical students as future physicians. Our goals are to design an educational intervention around critical appraisal in this context, assess the efficacy of the teaching methods, and investigate critical appraisal skills training to undergraduate medical students. We hypothesized that an educational intervention about critically appraising the scientific literature would advance students' medical skills in the EBM principle of critical appraisal.

Methods

Study design and applicants

This study was a prospective randomized control trial (RCT) conducted at Shiraz University of Medical Sciences (SUMS) in 2021; The undergraduate program of the Medical School at SUMS lasts seven years and includes horizontal integration of fundamental scientific courses and 36 months of clinical rotations.

In this study, the sample size was according to that in the study by Rod, S. Taylor, et al. Considering the average comparison formula of two independent communities, the sample size had been calculated at approximately to be 110 in each group in that study ( 4 , 13 ), with the confidence interval of 95%, standard deviation 1.56, and d about 0.097. Regarding at least 20% of the missing, this sample increased to 130 in each group. The control group did not receive any training, and the intervention group received training offline (multi-media) and then got training online (half-day workshop). Indeed, 124 of the students served as a control group, and 132 students served as an intervention group. We allocated students in each group based on the random numbers table. Finally, 256 sixth-year clinical students of Shiraz University of Medical Sciences answered all questions.

The CONSORT statement for randomized controlled trials was employed ( 14 ). Figure 1 shows a diagrammatic representation of this study design.

Figure 1. Diagram of the study design

In this study, the critical appraisal was taught offline by the instructor via multi-media, and then two weeks later, we set up an online workshop; this course was organized by the medical education department and focused on aiding the methodical examination of research findings in order to determine the study's validity, outcomes, and applicability to a given clinical circumstance. The instructor taught again critical appraisal based on three headings: 1- critical appraisal of an article about treatment, 2- critical appraisal of the article about diagnosis, and 3- critical appraisal of the article about prognosis; after that, the instructor answered the students' questions. A month later, we distributed the Critical Appraisal Skills Programme (CASP) questionnaire ( 4 ) online among both intervention and control groups to evaluate students' knowledge, attitude, confidence, and apprise skills. Worthy to note the pandemic prevalence, the questionnaires were distributed online among the participants, and then all the questionnaires were collected.

Permission of using original version of the CASP questionnaire was obtained via email from the author (Rod, S. Taylor) ( 4 ). This questionnaire was translated into Persian for the first time in Iran by Varai S, et al. ( 15 ). They had investigated the validity and reliability of this tool, the reliability of the questionnaire being obtained by the Richardson Index (0.75), and the validity being confirmed by some faculty members of Tehran Nursing and Midwifery School. In addition, after collecting, we compared the outcome between both groups to determine the effect of critical thinking education on medical students' abilities.

This questionnaire consisted of 18 multiple-choice questions that assessed knowledge, seven attitude statements, four critical appraisal skill, and six confidence statements questions. 'True,' 'False,' 'Don't know’ were all acceptable responses to the knowledge questions. Attitude announcements, confidence and critical appraisal skills were ranked on a five-point Likert scale. A five-point scale was used to determine which respondents strongly agreed to strongly disagreed ( 4 ). Moreover, the participants' demographic information such as age, gender, and so on was collected via the questionnaire.

Statistical analysis

Data analysis was performed, using SPSS software version 22 (IBM SPSS Statistics for Windows, version 22.0. Armonk, NY: IBM Corp, Chicago, USA, 2013). To analyze demographic information, we applied some descriptive statistics, namely, frequency, frequency percentage, mean and standard deviation, and to compare both control and intervention groups, we used an independent t-test and a Chi-Square test at a significance level of 5%. Moreover, we applied the Kolmogorov-Smirnov test to analyze the normality distribution of the questionnaire data, so all variables had a normal distribution (p>0.05). The effect size of the questionnaire dimensions was also calculated using Cohen's D, and it tells the researchers how meaningful the relationship between the groups is. It demonstrates the practical significance of a research outcome ( 16 ).

Ethical considerations

Our research was approved by the Ethics Committee of Shiraz University of Medical Sciences under the ethical code IR.SUMS.REC.1400.535. The students volunteered to participate in the study and complete the questionnaire. Plus, all of the participants were anonymous.

Inform Consent

Participants agreed to participate after being informed about the study's objectives and what participation entailed; researchers obtained written consent from each participant before conducting the study to uphold ethical guidelines.

Results

Description of participants' demographic information

The study population in this study includes sixth-year medical students at Shiraz University of Medical Sciences in 2021 who have taken critical appraisal online workshops. The sample size was estimated at 256, including 132 interventions and 124 controls. Generally, most of the students were male (N=164 (61.9%)). Our results showed that for all of the participants, the average hours to study was 1.89±2.85. Furthermore, the average hours to use the Internet was 2.16±3.84 every day. On average, they spend 2.64±2.1 hours studying academic articles weekly. There was no significant difference in demographic factors among the intervention and control groups (Table 1).

As can be seen in Table 2, in all dimensions of the critical appraisal skills questionnaire, the medical participants who received training offline and attended the critical appraisal workshop performed better than those who did not, and the differences were statistically significant. (P<0.05). The same results were seen in the mean values of the dimensions of the questionnaire (1- knowledge, 2- Attitude, 3- Confidence, 4- Appraise Skill).

The results of the Cohen's D effect size values showed that the knowledge and confidence dimensions are greater than 0.5; this meant a large effect. Regarding the attitude and appraises skill dimensions, the effect size was between 0.2 and 0.5, which meant a medium effect (Table 3).

Discussion

The results of this prospective RCT demonstrated that teaching critical appraisal skills through multi-media and a half-day CAS online workshop enhanced the students' ability to appraise information critically. This training method has positive effects on all aspects of critical appraisal. It is noteworthy to mention that we have found that training critical appraisal through this method has a more optimistic influence on knowledge of the principles and philosophy of evidence-based training and confidence than on the attitudes toward evidence or evidence seeking behavior and appraisal skills aspects.

In the study of Rod, S. Taylor et al., they indicated that the participants' attitudes didn't improve with critical appraisal training ( 4 ). We have found that critical appraisal training by multi-media and workshops can have positive, meaningful impacts on medical students' attitudes, with a medium effect size between the intervention and control groups. Our results may be due to the way of teaching, which was by multi-media, and the students had enough time to review it again.

The same results were reported in the study of Maden-Jenkins M. and Hyde C, et al. that showed critical appraisal training positively affected participants' attitudes ( 17 , 18 ).

Our study indicated that a critical appraisal workshop had a positive, significant effect on the knowledge of the intervention group. These results are in line with Hyde C, et al’s. and Thomas JD, et al’s. ( 18 , 19 ).

When comparing our results to previous studies, it must be pointed out that using multi-media in this study was significant and caused improvements in knowledge, attitude, confidence, and appraising skills. By transferring everything online, multi-media saves time, money, and resources. In terms of education, it makes learning accessible in every place ( 20 ).

The findings of this study afford only inadequate support for such teaching. However, it is essential to put this outcome in a proper educational setting. The half-day online critical appraisal skills workshop assessed in this trial has been widely spread, and its period and set-up are regular with many previous critical appraisal training interventions ( 21 ).

However, it is possibly unworkable to suppose that the half-day workshop estimated in this trial would result in deviations in professional performance. This is sustained by an enormous body of data and theory on altering professional training ( 22 ).

Critical appraisal projects in the undergraduate classroom can be effective strategies for student participation. As students learn to navigate scientific literature in a community of practice, it becomes less intimidating ( 23 , 24 ). It causes an increase in time and opportunities for continuous professional development and research, which may play a leading role in promoting the health of medical students ( 25 ). The most substantial part of critical appraisal training may modestly be to explain to members the accessibility of high-quality evidence. Additional discussion is consequently required about progressing critical appraisals skills training on the way to outcome such evidence and the starring role of health-care librarians and the new enterprises such as the National Electronic Library for Health. Several commentators have criticized previous evaluations of critical appraisal skills teaching for not using investigational strategies ( 26 - 28 ).

One of the strengths of the current study is that the design is RCT with the acceptable number of medical students in Shiraz Medical School, which is one of the major medical schools in the South of Iran. Another strength is that it was done based on the Oxford Regional Health Authority's Critical Appraisal Skills Program, which was designed using McMaster University's teaching methodologies in Canada ( 6 ). Self-directed learning, small group teaching approaches, and the significance of embedding education within the clinical decision-making process are major characteristics of the 'McMaster model ( 29 ).

Potential limitations of this study

As with all research, this study had restrictions. While specific improvements are statistically significant, we are unable to determine whether or not they are educationally meaningful. Another restriction is that the study is a single-center study; thus, generalizability is limited.

We might have prepared the students for the questionnaire during the workshop or through multi-media training, but, to overcome this limitation, we enlisted the assistance of an outside observer to supervise the educational parts.

Another drawback is that we did not use a pretest to compare the same students before and after training and instead used this test exclusively for posttests. However, pretest-posttest improves internal validity but sacrifices external validity to do so ( 30 ).

Conclusions

Teaching critical appraisal skills through multi-media and CAS online workshops to medical students effectively improves students' knowledge and confidence in appraising articles. This teaching also indicated a medium effect on students' attitudes and behavior. Further research is necessary to see whether this strategy leads to increased critical appraisal skills in actual clinical situations. Our findings can justify implementing critical appraisal skill teaching modules in the undergraduate medical education curriculum.

Acknowledgement

This study is a part of a thesis by the first author, Sarvin Sasannia, for obtaining a Medical Doctor's degree. This study was approved and financially supported by Shiraz University of Medical Sciences, Shiraz, Iran. We thank the faculty members and the medical students at Shiraz Medical School for participating in this study, and the Center for Development of Clinical Research of Nemazee Hospital, and Ms. Farhadipour for statistical assistance.

Authors' contribution

S.S, M.A, M.M, A.A, M.M, H.Z, F.V contributed to the conception and design of the work; the acquisition, analysis, or interpretation of data for the work. All Authors contributed in drafting and revising the manuscript critically for important intellectual content. All authors have read and approved the final manuscript and agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Conflict of Interest

None declared.

References

1. Djulbegovic B, Guyatt GH. Progress in evidence-based medicine: a quarter century on. Lancet (London, England).. 2017; 390(10092):415-23.
2. Al-Jundi A, Sakka S. Critical appraisal of clinical research. Journal of clinical and diagnostic research: JCDR. 2017; 11(5):JE01-05.
3. Horsley T, Hyde C, Santesso N, Parkes J, Milne R, Stewart R. Teaching critical appraisal skills in healthcare settings. Cochrane Database of Systematic Reviews. 2011; 2011(11):CD001270.
4. Taylor RS, Reeves BC, Ewings PE, Taylor RJ. Critical appraisal skills training for health care professionals: a randomized controlled trial [ISRCTN46272378]. BMC medical education. 2004; 4(1):1-10.
5. Sagheb MM, Amini M, Saber M, Moghadami M, Nabiei P, Khalili R, et al. Teaching evidence-based medicine (EBM) to undergraduate medical students through flipped classroom approach. Shiraz E-Medical Journal. 2018; 19(2)
6. Kong LN, Qin B, Zhou YQ, Mou SY, Gao HM. The effectiveness of problem-based learning on development of nursing students’ critical thinking: A systematic review and meta-analysis. International journal of nursing studies. 2014; 51(3):458-69.
7. Jaunoo S, Adams HL. A survey showing the need to incorporate the teaching of critical appraisal skills in the undergraduate medical curriculum. British Journal of Medicine & Medical Research. 2016; 14(12):1-8.
8. Halalau A, Holmes B, Rogers-Snyr A, Donisan T, Nielsen E, Cerqueira TL, et al. Evidence-based medicine curricula and barriers for physicians in training: a scoping review. International journal of medical education. 2021; 12:101-24.
9. McCall MC, Fanshawe TR, McCartney D, Young D, Nunan D, Heneghan C. Online supplementation for teaching evidence-based medicine: feasibility of a randomised-controlled trial. BMJ evidence-based medicine. 2021; 26(5):254.
10. Basheer A, Iqbal N, Prabakaran S, Simiyon M, Anandan V. Simulated Randomized Controlled Trial to Learn Critical Appraisal (SiRCA): A Randomized Controlled Study of Effectiveness Among Undergraduate Medical Students. Cureus. 2021; 13(11):e19946.
11. Gardeshi Z, Amini M, Nabeiei P. The perception of hidden curriculum among undergraduate medical students: a qualitative study. BMC research notes. 2018; 11(1):1-4.
12. Tabari P, Amini M, Arya N, Moghadami M, Khoshnood K, Shokripour M, et al. The role of educating health-care personnel in prevention, diagnosis, or treatment of COVID-19: A narrative mini review. Journal of Education and Health Promotion. 2021; 10(1):476.
13. Moghadami M, Amini M, Moghadami M, Dalal B, Charlin B. Teaching clinical reasoning to undergraduate medical students by illness script method: a randomized controlled trial. BMC medical education. 2021; 21(1):1-7.
14. Moher D, Schulz KF, Altman DG. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Clinical oral investigations. 2003; 7(1):2-7.
15. Varai S, Dehghan Nayeri N, Hosseiny SF, Ahmadi Chenari H. Critique and evaluation of published articles quality of Health Sciences qualitative Researches Journal based on CASP scale. Journal of Qualitative Research in Health Sciences. 2020; 8(4):9-15.
16. Cohen J. Statistical power analysis for the behavioural sciences. Routledge: New York; 1988.
17. Maden‐Jenkins M. Healthcare librarians and the delivery of critical appraisal training: attitudes, level of involvement and support. Health Information & Libraries Journal. 2010; 27(4):304-15.
18. Hyde C, Parkes J, Deeks J, Milne R. Systematic review of effectiveness of teaching critical appraisal. Database of Abstracts of Reviews of Effects (DARE): Quality-assessed Reviews [Internet]. Centre for Reviews and Dissemination (UK): York (UK); 2000.
19. Thomas JD, Cardinal BJ. Health science knowledge translation: Critical appraisal of online physical activity promotion material. Nursing & Health Sciences. 2021; 23(3):742-53.
20. Ellaway R, Masters K. AMEE Guide 32: e-Learning in medical education Part 1; Learning, teaching and assessment. Med Teach. 2008; 30(5):455-73.
21. Coomarasamy A, Taylor R, Khan K. A systematic review of postgraduate teaching in evidence-based medicine and critical appraisal. Med Teach. 2003; 25(1):77-81.
22. Bero LA, Grilli R, Grimshaw JM, Harvey E, Oxman AD, Thomson MA. Closing the gap between research and practice: an overview of systematic reviews of interventions to promote the implementation of research findings. BMJ. 1998; 317(7156):465-8.
23. Larsen CM, Terkelsen AS, Carlsen AF, Kristensen HK. Methods for teaching evidence-based practice: a scoping review. BMC medical education. 2019; 19(1):259.
24. Flynn MF. Developing critical appraisal and evidence synthesis skills in future microbiologists. FEMS microbiology letters. 2021; 368(18):1-3.
25. Omidifar N, Bafti AH, Shokripour M, Amini M, Erana-Rojas IE, Moghimizadeh M. Pathologists&#039; professional lifestyle: Excellence in practice, ethics, education, health promotion, and personal life. Journal of Education and Health Promotion. 2022; 11(1):97.
26. Taylor R, Reeves B, Ewings P, Binns S, Keast J, Mears R. A systematic review of the effectiveness of critical appraisal skills training for clinicians. Med Educ. 2000; 34(2):120-5.
27. Coomarasamy A, Taylor R, Khan KS. A systematic review of postgraduate teaching in evidence-based medicine and critical appraisal. Med Teach. 2003; 25(1):77-81.
28. Parkes J, Hyde C, Deeks JJ, Milne R. Teaching critical appraisal skills in health care settings. Cochrane Database of Systematic Reviews. 2001; 3:CD001270.
29. Milne R, Donald A, Chambers L. Piloting short workshops on the critical appraisal of reviews. Health Trends. 1995; 27(4):120-3.
30. Dimitrov DM, Rumrill JrPD. Pretest-posttest designs and measurement of change. Work. 2003; 20:159-65.