Introduction

The evaluation of the programs is mandatory in every faculty of medicine so that the authorities can assess the adequacy between the population needs, the curriculum, the pedagogical activities used, and the evaluation of competencies. For that reason, many faculties have been enrolled in evaluation processes mainly supervised by the World Health Organization (WHO), the International Conference of the Deans of Medical Faculties with French Expression (CIDMEF), and the World Federation of Medical Education (WFME). These organisms enable various faculties clearify their mission, objectives, organization, and social responsibility. Being enrolled into an accreditation or certification process provides the ground for evaluation by external evaluators. Even if the external evaluation is necessary and mandatory, an internal audit or evaluation represents a mainstay in the medical faculties. Some authors have published different recommendations dedicated mainly to external evaluators enrolled in an accreditation or a certification process ( 1 ), but few reports have been concerned with the internal evaluation made by the institution members ( 2 - 6 ). When dealing with the medical literature, evaluation or assessment of programs has been reported using non-consensual methods. The absence of consensus induces difficulties to ensure a permanent concordance between the curriculum and the population needs in of the issue of solving health problems. The majority of the authors used pre- and post-test evaluation, students’ reports, students’ satisfaction evaluation, self-assessment of students, or the students’ cognitive scores ( 2 , 3 , 6 , 7 ). In rare reports, the authors have performed an evaluation of a program based on the students, teachers and administrators’ evaluation ( 4 , 5 , 8 ). In this study, we aimed to assess the program of a theme taught in the second year of medical education by using a four-axis method published in the literature ( 9 ).

Methods

The four-axis method consists of the assessment of the pedagogical alignment, teaching methods and assessment tests, students’ and teachers’ attitudes and needs. This method was initially described by Tabbane C ( 9 ). To contextualize this method, the authors described a practical experience of assessment of a multidisciplinary theme (theme 16), which is taught during the second year of medical education. The first part of the evaluation process was dedicated to the assessment of the various axes described by Tabbane, and the second part highlighted the measures taken to address the weaknesses identified.

The study type

The authors performed a descriptive study to report a particular approach to evaluate a program.

The population

All the tutors involved in the teaching of the theme [78] and all the students [430] attending the courses were asked to fill out the satisfaction questionnaires related to specific aspects of their faculty. The tutors and the students who did not fill out the questionnaires were excluded from the study.

The characteristics of the theme assessed by the authors: Theme 16 is about the human body facing aggressions. It is taught in the second year of the first cycle of medical education. This theme integrates immunology, pathology, pharmacology, histology, physiology, and psychiatry. This theme is linked to theme 7, which is about the aggression and is taught in the first year of the first cycle of medical education. Theme 16 prepares the students to learn pathologies during the second cycle of medical studies, with an emphasis on the individual and community dimension of health. Seventy-eight teachers are involved in the teaching of this theme: 46 pathologists (2 teachers per session), 9 pharmacologists, 3 histologists, 16 immunologists and 1 psychiatrist. The time scale is divided into 26 hours for immunology, 18 hours for pharmacology, 24 hours for pathology, 8 hours for histology, 4 hours for physiology, and 2 hours for psychiatry. Four hundred and thirty students attended the course during the period of evaluation. During the study, almost all courses were recorded and made available on Moodle platform for the students’ use.

The intervention: Different steps of the four-axis method

1-Part 1 of the evaluation process: Evaluation of the four axes.

The evaluation of the theme program focused on 4 axes:

- Axis 1: The objectives of the medical training/job profile of the doctors: An analysis of the educational objectives according to their taxonomic level as well as an analysis of the relevance of the mini-learning modules and the suitability of the content of the mini-modules about the objectives was carried out.

- Axis 2: Teaching techniques/assessment methods: The evaluation of teaching techniques was done after viewing the distance learning videos which were available on the Moodle platform.

- Axis 3 and axis 4: Students and teachers: Two questionnaires for teachers and students were prepared using the recommendations published by Tabbane ( 9 ). These questionnaires focused on the following topics: the objectives of medical education, the curriculum integrating the content, evaluation, and the students’ and teachers’ needs.

2-Part 2 of the evaluation process: According to the results of the 4-axis evaluation, the necessary actions were taken.

The judgement criteria

1-Assessment of the teaching techniques and methods: Twenty videos of pathology teaching sessions, 20 videos of immunology teaching sessions, 10 videos of pharmacology teaching sessions, 2 videos of histology teaching sessions, 1 video of a psychiatry teaching session and 1 video of a physiology teaching session were randomly viewed disregarding the responsible teacher. The evaluation was based on a teaching evaluation guide published by the University of Toronto and took into account, among other things, the relevance of the objectives chosen during the interactive session (Clarification, Illustration, Application, Participation method), the use of bioclinical cases, the clarity of the messages delivered, the repetition of messages according to the needs of the students, the assessment of knowledge during the session (pre and post-tests), and the interaction between the teacher and the students ( 10 ). The checklist used by the authors is represented in Table 1. The maximum score according to this checklist was 8.

2-The questionnaires filled out by the students and the tutors: Both questionnaires are Likert-scale questionnaires and represented in Tables 2 and 3. As the questionnaires were previously published, no reliability or validity analyses were performed. All the teachers [78] involved in the teaching of the theme were asked to complete the questionnaires and all the students [430] attending the courses were also invited to answer the questionnaires. The evaluation questionnaire for teachers was sent to them during the validation meetings of the exam questions held on May 31 with the Department of Pathology, 08 June with the department of immunology and 9 June with the departments of pharmacology and histology. During these working meetings, the teacher coordinator of the theme presented to the various teachers some recommendations concerning the drafting of the questions (presentation of 10 minutes); then, various questions to be validated were discussed. The student questionnaire was sent by the Studies Directorate on June 25, following the exam held that morning.

Statistics

Because of the descriptive nature of the study, no statistical tests were used. Qualitative data were reported using percentages.

Ethical considerations

In this study, ethical approval was not needed. Participants were informed about the aim of the study, anonymity, and the dataset generated; they were also insured that they can withdraw from the study at any time in this way, informed consent was obtained from each respondent.

Results

1-Part 1: Four-axis-based evaluation

- Axis 1: All mini-modules met the structure recommended by the faculty consisting of pre-requisite, educational objectives, references, and self-evaluation tests. The physiology mini-module did not contain educational objectives. The details of the analysis of the objectives taxonomy are represented in Table 4.

- Axis 2: The average score of the teaching sessions was 3, ranging from 2 to 6.

Regarding the methods of evaluation and after evaluation of the tests of the previous year, the pedagogical alignment (taxonomic level of the objective/ question asked) was ensured in 50% of the questions of pathology, 80% of the questions of pharmacology, 40% of immunology questions, 100% of psychiatry questions, and 100% of histology questions. The lack of alignment was mainly due to the formulation of questions evaluating levels 2 or 3 objectives when only level 1 objectives were formulated and made available into the mini-learning module. Nevertheless, the answers to various questions posing a pedagogical alignment problem were included in the mini-learning module.

- Axis 3 and axis 4: 31 teachers and 88 students filled out the questionnaires. The responses were analyzed under the above headings.

The objectives of medical training: 67.7% of the teachers did not read or reread the national doctor’s job profile for at least the last five years. This profile was taught to our students as part of Theme 1 and was presented to all the teachers as part of the educational workshop entitled “Problematic part of the medical training”. This workshop is mandatory for all first-year teachers. 51.1% of the students remembered the national doctor’s job profile. 64.5% of the teachers interviewed participated in the development of the educational objectives. 67.7% of the teachers found the objectives unambiguous for students. Also, 48.8% of them found the objectives of their specialty motivating. 54.6% of the students did not read the objectives for each course. Among students who claimed to have read the objectives, 55% found that the pathology objectives were clear and unambiguous.

- Data on content, teaching methods, and evaluation of theme 16: 74.2% of the teachers reported that they knew the whole program of theme 16 in order to better integrate their own teaching. 74.2% of the teachers believed that their teaching met the competencies listed in the job profile. 32.3% of teachers felt that they did not establish the link between the clinic and the scientific aspects of diseases in their CIAP sessions. According to 93.5% of the teachers, the mini modules of their disciplines are written with objectives, basic texts, and assessment tests. On the other hand, 53% of the students perceived that the mini modules were incomplete, and did not contain educational objectives, basic documents and self-assessment tests. 48.4% of the teachers did not systematically use bioclinical cases involving interactions between students. 42.4% of the students believed that the teaching sessions did not introduce practical examples giving rise to interactions between them. 71% of the teachers did not prepare the pre-tests and post-tests to evaluate their teaching. 48.4% of the teachers were not aware of the docimological data relating to their questions included in the final exam. 89.4% of the students felt that the hour volume and its distribution did not leave them enough time for their self-directed learning.

- Students’ views and needs: 69.4% of the students reported they read the minimodules before the teaching sessions. Apart from the mini module, the main sources of information used by the students were YouTube channels and sites of other medical schools with different videos. Among the improvements requested by the students regarding the content, teaching methods, and evaluation of the theme, the students suggested reviewing the courses, treating as many clinical cases as possible during teaching sessions, integrating more applications, exercises and clinical cases into CIAP sessions, reviewing the educational objectives, and ensuring the pedagogical alignment with the assessment.

Axis 4: The teachers’ views and opinions: All the teachers rated their motivation between 6 and 10 according to the scale given to them. The teachers estimated that they used the concepts learned in the educational workshops organized by the faculty, in a range of 5 to 10. Most teachers reported it was difficult to apply CIAP sessions using bioclinical examples in their teaching. In addition, they deplored the formulation of level 1 educational objectives which can only be evaluated with questions relating to the simple memorization of some concepts.

2- Part 2: The actions taken to correct the weaknesses identified:

The results of this evaluation were communicated to the coordinators ofvarious departments to encourage them to apply the necessary recommendations within their specialties. In addition, an invitation to a work meeting was sent to the teachers affiliated to the 6 departments included in the teaching of this theme. Teachers wishing to strengthen their teaching skills registered via a Google Forms, and a working meeting was organized on 26/02/2022 within the Faculty of Medicine. During this work meeting, the main results of this evaluation were presented, and the teachers present were called upon to work in small groups on the main problems mentioned above. As part of various exercises, they were asked to assess the structure and relevance of their self-learning mini modules, to present in plenary different bioclinical cases they use for their teaching during their CIAP sessions, to script their CIAP sessions after a careful choice of the educational objectives according to their relevance to the job profile and to verify the educational alignment between the educational objectives of their self-learning mini-modules, the teaching techniques, and their evaluation questions.

At the end of the meeting, documents prepared by the pedagogical committee (the self-directed-learning mini module, the structure of a self-learning mini module, the analysis of a self-learning mini module, the criteria for evaluating the relevance of a course, and the checklist for preparing a CIAP session) were sent to all the teachers to consolidate their learning.

Discussion

The authors made an attempt to evaluate a multi-disciplinary theme taught during the second year of medical education. This evaluation was based on the methodology published by Tabbane ( 9 ). It consisted of a 4-axis-based analysis. Axis One was dedicated to the evaluation of the educational objectives and their pedagogical alignment with the national doctors’ job profile and the assessment, axis 2 focused on the evaluation of the teaching techniques, and axis 3 and axis 4 focused on the students’ and teachers’ views and potential needs. According to this evaluation process, the strengths of this theme consisted of the integration of several specialties allowing the students to integrate the individual, community, and anthropological dimensions of the human body facing aggression. Weaknesses included the lack of homogeneity of the teaching methods and techniques showing the teachers’ problems in integrating the clinical data into their teaching so that they can allow the students to assimilate fundamental data. Besides, 80% of the educational objectives were of level 1: the students were limited to a simple memorization of the information; there was pedagogical alignment between objectives; teaching and assessment techniques were not always respected; the questions asked during the evaluation were of a higher level compared to the objectives prompting their revision; and the lack of knowledge of the job profile by teachers can lead to a lack of relevance in the educational objectives. The opportunities were presented through the accreditation process by the CIDMEF undertaken by the dean which can promote the quality of this theme. The threats were represented by the lack of federation of various coordinators around the final product, which is the students. The main areas of revision that seemed necessary to be undertaken consisted of a revision of the content of the mini modules, asking the teachers to multiply practical examples, reinforcement of the teachers training in the preparation of educational objectives, optimization of the sources of information for students and promotion of the validation of the questions within different departments included. The major strengths of this study consist of taking into account an objective analysis of the self-directed learning mini-modules, teaching techniques, evaluation of the students’ competencies and the students’ and teachers’ opinions and needs. The areas of discrepancies and contradictions were taken as measures to correct during the workshop planned at the end of the evaluation process. In the literature, many authors evaluated medical programs based on pre- and post-test studies ( 6 ), attitudes or qualitative studies, and students’ scores ( 7 ). The four-axis method seemed to be more efficient than taking into account the students’ scores alone or their self-perception. Besides, our approach encouraged the teachers to think about their teaching practice critically. This exercise could increase their critical thinking practice. There are many definitions in the literature about critical thinking. The mostly agreed one was that of Lipman ( 11 ). As he dfines it, critical thinking helps one reach a good judgment. Kpazai et al. introduced a pedagogical analysis of critical thinking developed aiming to assess this competence in health and physical education teachers. They concluded that teachers with a high critical thinking potential were more able to transmit this important competency to students ( 12 ). The major limitation of the four-axis method was that it did not consider the perception of the educational environment by the students. A learning environment has been defined as the social, psychological, and pedagogical settings where learning occurs ( 13 ). It plays a central role in the learning process according to some authors. According to Chellaiyan et al., understanding the learning environment determinants is necessary for improving the lacune of an institution ( 14 ). Besides, some authors reported that health educational institutes in the Western world were more able to offer student-friendly approaches to education than Asian and Middle Eastern institutions ( 15 , 16 ). Making emphasis on the learning environment is crucial because it was reported to be correlated to the students’ scores ( 14 ); besides, it highlights an authentic social contract with the students ( 1 ). Another limitation of this study was represented by the lack of similar experiences inducing a limitation in the comparison with eventual previous studies.

Conclusion

The four-axis method, used to assess a medical program, was unique due to the fact that it focused on not only an objective evaluation of the teaching techniques and the evaluation tests but also the students’ and teachers’ opinions and needs. It revealed the heterogeneity of the teaching methods and techniques showing the teachers’ difficulties to integrate clinical data in their teaching so that the students are allowed to assimilate fundamental data. The main limitation of this approach lies in the fact that it did not take consider educational environment, which could impact the students’ performances. The educational environment assessment represents a further field of research.

Conflict of interests

The authors declare no conflicts of interest.

References

1. Des Marchais JE, Rothman A. Évaluation externe et formative de programme, étapes de la méthode. Pédagogie Médicale. 2001; 2:213-21.
2. Pesesse L, Dozo BO, Quatresooz P, Weatherspoon A, Multon S, Pirotte M, et al. Designing a Massive Open Online Course (MOOC) in face-to-face sessions, A blended design to teach practical histology. Morphologie. 2023; 107(359):100604.
3. Ozone S, Haruta J, Takayashiki A, Maeno T, Maeno T. Three-year evaluation of a program teaching social determinants of health in community-based medical education: a general inductive approach for qualitative data analysis. BMC Med Educ. 2023; 23:1.
4. Cochetel O, Oupathana B, Deschamps JP. Evaluation of a medical teaching programme in public health in Laos. Sante Publique (Paris).. 2005; 17:95-107.
5. Bonnaud A, Pottier P, Barrier JH, Dabouis G, Planchon B, Mouzard A. Évaluation d’un enseignement concernant la relation médecin-patient auprès des étudiants de deuxième et troisième années des études médicales à la faculté de médecine de Nantes. Pédagogie Médicale. 2004; 5:159-66.
6. Xu G, Zhao C, Yan M, Zhang X, Zhu L, Liu J, et al. Evaluating the effectiveness of a new student-centred laboratory training strategy in clinical biochemistry teaching. BMC Med Educ. 2023; 23:11.
7. Karimiankakolaki Z, Mazloomy Mahmoodabad SS, Kazemi A. Designing, implementing and evaluating an educational program regarding the effects of second-hand smoke in pregnancy on the knowledge, attitude and performance of male smokers. Reprod Health. 2023; 20(1):82.
8. Guedjati MR. Évaluation des programmes d’études de la première et de la deuxième années de médecine-Faculté de médecine de Batna. Université Hadj Lakhdar Batna, Faculté de Médecine: Algeria; 2013.
9. Tabbane C. Evaluation d’un programme d’enseignement médical et des conditions de sa réalisation, Eléments d’introduction aux ateliers de pédagogie médicale. Centre de Publication Universitaire: Tunis; 2000.
10. Gregor-Greenleaf E, Burnett M, Gravestock P. Developing & Assessing Teaching Dossiers: A guide for University of toronto faculty, administrators and graduate students. The Centre for Teaching support and innovation (CTSI), University of Toronto: Toronto; 2017.
11. Lipman M. Moral education higher-order thinking and philosophy for children. Early Child Dev Care. 1995; 107:61-70.
12. Kpazaï G, Daniel MF. Manifestations of Critical Thinking in Health and Physical Education Teachers: An Examination of Three Case Studies. PHENex Journal. 2011; 3:1-15.
13. Hutchinson L. Educational environment. BMJ. 2003; 326:810.
14. Chellaiyan VG, Bhushan K, Maruthupandian J, Gupta S, Liaquathali F, Britto J. Maedica-a Journal of Clinical Medicine MAEDICA-a Journal of Clinical Medicine Perception of the Educational Environment and Its Impact on the Academic Performance of Medical School Students: a Cross-Sectional Study. Maedica (Bucur).. 2023; 18(1):80-7.
15. Palomo-López P, Becerro-de-Bengoa-Vallejo R, Calvo-Lobo C, Tovaruela-Carrión N, Rodríguez-Sanz D, Elena Losa-Iglesias M, et al. Student perceptions of the education environment in a Spanish medical podiatry school. J Foot Ankle Res. 2018; 11:14.
16. Bakhshialiabad H, Bakhshi G, Hashemi Z, Bakhshi AH, Abazari F. Improving students’ learning environment by DREEM: an educational experiment in an Iranian medical sciences university (2011–2016).  BMC Med Educ. 2019; 19:397.