Introduction

Learning in the clinical environment is often described as experiential, whereby Kolb’s learning theory considers the transformation of an experience to a construct that can be further tested and refined ( 1 ). Simulation-based education are activities that replicate real events and give opportunities for learners to participate in either an active or passive role. There are simulated experiences in healthcare education that can provide levels of fidelity (from low to high) and modalities that can include patient manikin, virtual reality, actors, and simulation trainers ( 2 ). For example, healthcare educators might aim to mimic a clinical encounter in simulation to provide a learning experience with the use of immersive 360-degree video (360). Innovative Technology Enhanced Learning (TEL) provides an opportunity to learn from a digital resource ( 3 , 4 ), which can be both remote from origin and experienced anytime.

As new TEL expands rapidly, it allows learners to consume digital material in various formats asynchronously to others and via different electronic devices. Electronic learning is the delivery of knowledge and skills via digital resources ( 5 ). The E-Learner typically navigates these mediums in a capacity for autonomous learning that is part of a goal orientated self-actualised behaviour ( 4 ). Health care professionals (HCP) can access this diverse content from mobile devices or computers. In a study conducted in a Canadian medical school reported over 85% of medical students regularly using mobile phones for education purposes ( 6 ).

A United Kingdom National Health Service review entitled ‘preparing the healthcare workforce to deliver the digital future 2019 review’ was predicated on pre-suppositions. In particular,

‘There is remarkable potential for digital healthcare technologies to improve accuracy of diagnoses and treatments, the efficiency of care, and workflow for healthcare professionals, but implementation must only be carried out when there has been robust clinical validation.’ ( 7 ).

Immersion can be defined as a psychological state of engagement with a task or medium that varies in depth ( 8 ), and where the presence or connection to the physical world is inversely associated with task engagement ( 9 ). Technology allows viewers to experience a recreation of the clinical environment whilst remaining remote or asynchronous to the event. For example, a 360 video of a clinical consultation or a virtual reality (VR) simulated patient ( 10 ). The level of interactivity with the learning environment can exist on a continuum from less to more interactive, with 360 being a passive consumption of media ( 11 ). The designer of the 360 resources can, however, edit and adapt to learning needs of a group and provide financial benefit through scalability ( 12 ). A significant number of authors conclude that immersion was an important feature of education using 360 or VR ( 13 - 17 ).

Learning, in part, is determined by a person’s motivation states and how their self-image projects a readiness to learn. Motivation to learn holds intrinsic value with a personal self-directed pay off. This being self-governed by a need know: why, what and how? ( 18 ) These assumptions to principles of learning forms part of the andragogy of learning. Learning theory that arguably derives a curriculum from HCP self-actualisation and competence is based on constructs and assumptions, creating a challenge to measure and understand the homogenous and heterogenous concepts within these domains of learning. Motivational states relate to numerous learning approaches researched ( 4 , 19 - 21 ) and have been described as intrinsic and extrinsic in nature.

Healthcare educators would benefit from understanding how TEL might influence a user’s immersion in or engagement with learning media and thus consider how this affects learning. Given a learner’s motivational state could influence the user’s engagement with technology and content, there is implication that quantifying this alongside immersion will enable greater understanding. Having a validated measure that reliably quantifies user experience whilst consuming medical based media would allow both learner and educator to reflect on how we might learn. Currently, there is a lack of standardised measures that allow assessment of engagement and comparison to different media in medical education ( 22 ).

Although we can assume that immersion is similar in using TEL in medical education to that of gaming, there are underlying motivational and outcome differences. This highlights the importance of testing metrics for their validity for use in the clinical environment.

This study aims to assess engagement and motivation in this context, and describes the adaptation and validation of a questionnaire for immersive technology in medical education. Additionally, authors will aim to demonstrate reliability to ensure reproducibility.

Review of existing measures for engagement in video games and motivation

User immersion and related concepts have been a focus for video games researchers for several decades, with numerous measures being developed ( 23 ). Researchers in this area often regard immersion as a state that incorporates enjoyment and real-world dissociation to heighten gaming experience. Jennett, et al. ( 9 ) conceptualised that immersion encompasses dimensions of presence, cognitive absorption and flow. These factors are applicable to non-participatory video as though, a linear media, users can experience the 3 factors ( 24 ). Other psychometrics exist that measure, for example, Core Elements of the Gaming Experience. However, the domains of interest are too game-centric as feature scales on puppetry, control, and gameplay ( 25 ). Thus, less suitable to this study.

The IEQ benefits from quantifying emotional connection, dissociation from the real-world environment, and resumed task engagement. The objective measure was constructed by comparing a control group and a gaming group and measuring time to re-engagement following the end of the task. Additionally, eye movement tracking was used to ensure participant remained focus on the gaming task as a method of reliability check. Furthermore, measure results were matched to show high correlation with total immersion score of IEQ and time to re-engage. Most IEQ questions are not specific to media content and are transferable as it is a measure of an experience.

A questionnaire that evaluates the internal motivations orientated to a task was developed by Ryan ( 26 ) and the full instrument is a 22-item measure that includes the domains of: perceived choice, pressure, enjoyment or interest, effort, and value. The experimental work established that the enjoyment subscale had greatest relevance to intrinsic motivation and hence, has the greater proportion of questions. The value subscale has been used in internalisation studies and reflects aspects of self-regulation. Both the full measure and abbreviated versions have shown validity in numerous studies ( 27 - 29 ). The Intrinsic Motivation Inventory (IMI) was abridged (AIMI) to a 12-item measure that included subscales on enjoyment and value in this study with the assessment of reliability.

Methods

Adapted immersive experience questionnaire development

The multi-dimensional measure of engagement is based on existing video game research of Jennett, et al. and their development of the Immersive Experience Questionnaire (IEQ) ( 9 ) and adapted to a simulated HCP interaction: the adapted immersive experience questionnaire (AIEQ). Similar work on validating IEQ to film incorporates the principles explored with relationship to media on differing screen size ( 30 ). As adaptations to the questionnaire are required for use in the clinical setting a process of content validity is needed. Prior exploratory factor analysis on IEQ ( 9 , 30 ) informs of existing construct validity.

The research group in 360-degree video at Great Western Hospital (GWH) undertook review of 31 questions of original IEQ to correct game-specific words and alter tense for correct grammar relating to watching a clinical consultation. Eight questions were not applicable to the study, for example, ‘how much did you want to win the game?’. Additionally, if rewording was not possible, then a replacement question would keep the theme applicable to immersive media. Following this work, the AIEQ consisted of 23 items using a 1-5 Likert scale and 1 item using a 1-10 Likert scale (see table 1). The AIEQ was piloted with a small sample of 4 medical students.

Comparison of experience following consumption of the same media on two different technologies was the validation experiment analysing the effect of immersion and tool development.

Reliability and validity analysis

After the development of the AIEQ the scores were analysed to test the internal validity of the questionnaire using Spearman’s rank correlation. Also, reliability analysis with comparing Cronbach alpha was undertaken to ensure questionnaire reconstruction maintained a high reliability that Jennett, et al. described ( 9 ).

Study design

This study used a between-participants design, with two conditions: 1. Viewing a 2D medical consultation on a laptop screen, and 2. Viewing the same consultation in 360 using a Samsung Galaxy Gear VR, where participants had the freedom to look around. The dependent variables were immersion score (assessed by the AIEQ) and intrinsic motivation score (assessed by the AIMI).

Participants

The measures were completed by 40 participants. This was a pool of different HCP, including: medical students, junior doctors, and physician associate students, see table 2. These were recruited from GWH, Swindon, and University of West of England. Sampling of participants took place during core clinical teaching sessions. Thirty were female (75%), and 10 were male (25%). Twenty-eight participants (72%) were in the 20-25 age range, and the remaining 11 participants (28%) were aged 25-30.

Inclusion criteria:

1. • Age ≥18
2. • Student status as a healthcare professional
3. • Qualified healthcare professional and undertaking further education related to clinical training.

Exclusion criteria:

1. • Pilot participants who viewed the video.
2. • Qualified status in healthcare setting and no longer in a training programme.

Computer generated randomisation occurred to allocate to 2 groups: 360 arm and conventional 2D arm. This was a simple randomisation following consent with the participants initially blinded to video format.

Flow diagram of parallel randomised trial CONSORT diagram outlines the process, in Figure 1.

There was no reward for inclusion in the study.

Figure 1. CONSORT flow diagram of trial

Ethical Consideration

University of Bath psychology departmental ethics committee approved the project in 2019 (ref 18-092). Participant information was anonymised and stored securely in accordance with the General Data Protection Regulations United Kingdom (GDPR).

Materials

Stimuli for the study consisted of two versions of a consultation between a General Practitioner (GP) and a trained clinical actor presenting with abdominal pain in a GP surgery. The first was a 360-degree video of this session, recorded on a 360-camera positioned on a tripod centrally in the room. This would give the viewer a virtual position placed in the middle of the room. The GP moved around the room during the consultation. The second was a 2D version of the same session, created from the original 360-degree recording: the video was edited to insert a title, and to create a 2D version of the consultation mapping the movements of the actors to mimic camera panning to give content similarity between formats. Both videos were approximately seven minutes in duration.

Data were collected from the participants using an online form, which consisted of participant information followed by digital consent form and AIEQ and AIMI. A Samsung Gear 360 VR headset was provided with a laptop to complete measures following watching the consultation and to display the flat screen (2D) film version.

Procedure

The experiment was conducted in a partitioned lab that divided a simulation suite to provide a quiet area with no visual or sound distractions. A researcher facilitated the experiment, which began by explaining the procedure to the participants and helping them familiarise themselves with the Samsung Gear VR controls. The headset controls allowed users to pause the video using a button on the side. This functionality was available to those viewing the consultation in 2D using the laptop. The researcher also explained that nausea wearing a VR headset can occur and demonstrated how to remove the device. Before commencing, the participants completed a consent form. The participants assigned the 360-video condition were seated on a swivel chair away from any obstacles, and participants assigned the 2D condition were seated in front of the laptop. The participants then viewed their assigned video condition and completed the AIEQ and AIMI immediately afterwards with data recorded on Google survey tool with a provided laptop. Finally, they completed a questionnaire to collect demographics of education background and whether they were an HCP student or qualified and in training. One member of the research team remained present behind the partition to be available for any difficulties experienced.

Statistical analysis plan

A test for kurtosis and histogram plotting determined normality of data. From previous studies non-parametric dataset is predicted using IEQ. Likert scales will be treated as continuous data with Wilcoxon’s signed rank for non-parametric data, Spearman’s rank for correlation, and Cronbach’s Alpha for reliability will be used for statistical analysis of scores. These will be undertaken using StatsDirect statistical software (version 3.3.5).

StatsDirect (version 3.3.5) sample size calculation is based on prior work on IEQ with estimated difference in means of groups of 10.00 and estimated standard deviation of 10.00. ( 30 ) Additionally, a probability of 80% of correctly detecting a real effect (power) and 5% probability of incorrectly rejecting hypothesis (alpha) were assumed. The unpaired two sample estimates a minimum sample of 17 in each group.

Results

Descriptive findings

No missing data was found, and all 40 participants’ responses were analysed. Through process of computerised random group allocation, 17 were randomised to the 2D arm and 23 to 360 arm.

The 40 responses for the AIEQ were scored with score inversion of the counter negative questions. A test for kurtosis and histogram plotting implied non-normally distributed data. The final sum was calculated with a potential maximum of 120.00, total immersion score (TI Score). 2D arm median TI score was 65.00 (IQR=59.00-70.00) and 360 median TI score 80.00 (IQR=75.00-85.00). Maximum TI score for 2D was 83.00, compared with 92.00 for 360 group. See table 3 for descriptive findings.

Outcomes

A Wilcoxon signed-rank test indicated that the immersion scores were statistically higher in those experiencing 360 (p<0.05), with a median difference of 14.50 (95% CI 6.50-22.00). A positive Spearman Rank correlation existed between the TI score and total score of the AIMI in both groups (r_s =0.88, n=17, p<0.001) as represented in Figure 2.

Figure 2. Scatter plot representing variable of total immersion score and total motivation score

Additionally, a positive Spearman Rank correlation existed between Q.21 that asked users to rate their immersion, Likert score 1-10, with TI score (r_s =0.87, n=40, p<0.001). This positive relationship establishes a degree of criterion-related validity on AIEQ for an index of immersion.

Measure Reliability

To screen for internal consistency reliability within 23-item AIEQ an overall alpha was calculated with a value of 0.91, which represents high consistency as above accepted 0.8. Individual questions were analysed for possible deletion; however, the maximum change for any question removal was -0.008, which would not significantly improve the alpha score. Participant numbers were too small to conduct an exploratory factor analysis to investigate if a predictive subset of variables existed.

AIMI alpha value was also high (α= 0.95), which shows the 12-item measure to be of good reliability. Again, the highest change whether any question to be removed was -0.008 and this would have no statistical impact on the alpha score found. Again, participant numbers were too low to analyse any meaningful subsets of variables.

Discussion

Innovative technology that transports the user into a clinical setting creates an increase engagement with the subject matter. AIEQ demonstrated a significant score difference when audiences viewed the same clinical consultation on a 360 headset. Factors influencing this included the real-world dissociation and captivation that 360 offers over conventional 2D audio-visual experiences. Rigby, et al. found similar findings when participants watched films on differing screen sizes with the larger screens demonstrating greater participant immersion ( 30 ).

In a study that utilised 360 headsets to measure the effects of motivation and competency found higher motivation and self-reported learning competency (p<0.05) ( 31 ). The tools to measure these findings reported no testing of the validity or reliability of the instruments. Similarly, in studies utilising 360 for simulation of palliative care, paediatric scenarios, and situational awareness indicated high degrees of immersion and positive student experience ( 32 - 34 ). However, in each of these studies the measures to capture the experience lacked a process of assessing the suitability and reliability of the questions asked.

Higher levels of motivation correlated with higher immersion scores in this study and this could represent a paradox, in that, does increasing immersion motivate a learner or vice-versa. Further studies to evaluate participant experiences in the motivational and engagement states of individuals in learning could be valuable to explore this. This study considered values and enjoyment attributable to intrinsic motivation drivers and as we aim to encourage group learning and working, scope exists to consider evaluating extrinsic motivational factors and communities of practice theory ( 4 ).

Although statistically significant findings were demonstrated, a limitation to this study was the low participant numbers and caution applied to how results can be interpreted. This study represents exploratory work that begins the foundations of creating methods for capturing user experience with visualisation technology. Unequal group sizes were a product of randomisation, which may have influenced the results. However, unequal group size in randomised trials can be expected and moderate disparities are accepted ( 35 ).

Interpreting AIEQ with differing TEL should note that users of new technology can experience a feeling and emotion of awe ( 36 ). This can influence users’ responses with a positivity towards using new technology, which may be temporary and mask any opposing thoughts.

Future work could explore this by including qualitative information that would supplement the data collection on any metrics ( 6 ).

Given the lack of standardisation in measures to quantify user experience with TEL or participants in simulation ( 37 ) this study evidences that existing validated measures can be adapted for clinical use. Learning and video gaming can be considered as not mutually exclusive as the gaming research community have shown learning occurs whilst gaming ( 38 ). Future work could expand on this by performing more in-depth validation exercise of the AIEQ, such as a factor analysis as in the original IEQ and the version adapted for video media. While the current study demonstrates this measure as appropriate for clinical use, this could lead to a greater understanding on the underlying structure of the measure providing additional reassurances of its validity.

Conclusion

TEL will continue to develop and advance into medical education and how we integrate this in HCP curricula matters. Possessing a greater understanding on how these new technologies affect the user experience can bring us a step closer to interpretation within the context of the complexity of learning.

This study evidences the reliability and validity on adapting instruments to measure engagement and immersion.

Medical education could benefit from borrowing and adapting more of the existing measures employed in assessing participant engagement and other learner domains as this can have numerous applications for learners, educators and developers.

Authors' contribution

Ch.J, J.M.R contributed to the conception and design of the work; the acquisition, analysis, or interpretation of data for the work. All Authors contributed in drafting and revising the manuscript critically for important intellectual content. All authors have read and approved the final manuscript and agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Conflict of Interest:

None declared.

References

1. Experiential learning: experience as the source of learning and development. Englewood Cliffs. Prentice-Hall: London; 1984.
2. Forrest K, McKimm J. Healthcare simulation at a glance. John Wiley &amp; Sons: Hoboken, NJ; 2019.
3. Moore JL, Dickson-Deane C, Galyen K. E-Learning, online learning, and distance learning environments: Are they the same?. The Internet and Higher Education. 2011; 14(2):129-35.
4. Swanwick T, Forrest K, O'Brien BC. Understanding medical education [electronic resource]: evidence, theory, and practice [3rd]. Hoboken. WILEY Blackwell: NJ; 2019.
5. Kumar Basak S, Wotto M, Bélanger P. E-learning, M-learning and D-learning: Conceptual definition and comparative analysis. E-Learning and Digital Media. 2018; 15(4):191-216.
6. Wallace S, Clark M, White J. ‘It&#039;s on my iPhone’: attitudes to the use of mobile computing devices in medical education, a mixed-methods study. BMJ Open. 2012; 2(4):e001099.
7. Topol E. Preparing the healthcare workforce to deliver the digital future. Final Report February 2019 - A Call For Evidence. Health Education England: England; 2019.
8. Kapralos B, Uribe-Quevedo A, Dubrowski A. Immersive Technologies for Medical Education; Encyclopedia of Computer Graphics and Games. Springer International Publishing: Cham; 2017.
9. Jennett C, Cox AL, Cairns P, Dhoparee S, Epps A, Tijs T, et al. Measuring and defining the experience of immersion in games. International Journal of Human-Computer Studies. 2008; 66(9):641-61.
10. Patel D, Hawkins J, Chehab LZ, Martin-Tuite P, Feler J, Tan A, et al. Developing Virtual Reality Trauma Training Experiences Using 360-Degree Video: Tutorial. J Med Internet Res. 2020; 22(12):e22420.
11. Roussou M, Oliver M, Slater M. The virtual playground: an educational virtual reality environment for evaluating interactivity and conceptual learning. Virtual Reality. 2006; 10(3):227-40.
12. Sandars J. It appeared to be a good idea at the time but … A few steps closer to understanding how technology can enhance teaching and learning in medical education. Med Teach. 2011; 33(4):265-7.
13. Patel DAO, Hawkins JAOX, Chehab LAO, Martin-Tuite PO, Feler JAO, Tan AAO, et al. Developing Virtual Reality Trauma Training Experiences Using 360-Degree Video: Tutorial. J Med Internet Res. 2020; 16:22.
14. Zulkiewicz B, Boudewyns V, Gupta C. Using 360-degree video as a research stimulus in digital health studies: lessons learned. JMIR Serious Games. 2020; 8(1):e15422.
15. Pieterse AD, Huurman VAL, Hierck BP, Reinders MEJ. Introducing the innovative technique of 360° virtual reality in kidney transplant education. Transpl Immunol. 2018; 49:5-6.
16. Dyer E, Swartzlander BJ, Gugliucci MR. Using virtual reality in medical education to teach empathy. JMLA. 2018; 106(4):498-500.
17. McGrath JL, Taekman JM, Dev P, Danforth DR, Mohan D, Kman N, et al. Using Virtual Reality Simulation Environments to Assess Competence for Emergency Medicine Learners. Academic Emergency Medicine. 2018; 25(2):186-95.
18. Knowles MS. The modern practice of adult education; andragogy versus pedagogy, by Malcolm S. Knowles, Association Press: New York; 1970.
19. Mann KV. Theoretical perspectives in medical education: past experience and future possibilities. Med Educ. 2011; 45(1):60-8.
20. Cassidy S. Learning Styles: An overview of theories, models, and measures. Educational Psychology. 2004; 24(4):419-44.
21. Artino AR, Konopasky A. The Practical Value of Educational Theory for Learning and Teaching in Graduate Medical Education. J Grad Med Educ. 2018; 10(6):609-13.
22. Schifferdecker KE, Berman NB, Fall LH, Fischer MR. Adoption of computer-assisted learning in medical education: the educators&#039; perspective. Med Educ. 2012; 46(11):1063-73.
23. Sweetser P, Wyeth P. GameFlow: A Model for Evaluating Player Enjoyment in Games. Computers in Entertainment. 2005; 3:3.
24. Busselle R, Bilandzic H. Measuring Narrative Engagement. Media Psychology. 2009; 12(4):321-47.
25. Calvillo Gamez E, Cox A. Chapter 4, Assessing the Core Elements of the Gaming Experience. Springer-Verlag London Limited: London; 2010.
26. Ryan RM. Control and information in the intrapersonal sphere: An extension of cognitive evaluation theory. Journal of Personality and Social Psychology. 1982; 43(3):450-61.
27. Ryan RM, Mims V, Koestner R. Relation of reward contingency and interpersonal context to intrinsic motivation: A review and test using cognitive evaluation theory. Journal of Personality and Social Psychology. 1983; 45(4):736-50.
28. McAuley E, Duncan T, Tammen VV. Psychometric properties of the Intrinsic Motivation Inventory in a competitive sport setting: a confirmatory factor analysis. Res Q Exerc Sport. 1989; 60(1):48-58.
29. Plant RW, Ryan RM. Intrinsic motivation and the effects of self-consciousness, self-awareness, and ego-involvement: An investigation of internally controlling styles. Journal of Personality. 1985; 53(3):435-49.
30. Rigby JM, Brumby DP, Gould SJJ, Cox AL.Paper presented at: ; 2019.
31. Sattar MU, Palaniappan S, Lokman A, Hassan A, Shah N, Riaz Z. Effects of Virtual Reality training on medical students&#039; learning motivation and competency. Pakistan Journal of Medical Sciences. 2019; 35(3):852-7.
32. Mason T. G627(P) Is virtual resus training the future? Can 360 degree ‘immersive video’ engage paediatric trainees safely in life support training? – A project as part of the south west return to paediatric training course (SWRPT). Archives of Disease in Childhood. 2019; 104(Suppl 2):A254.
33. Taubert M, Webber L, Hamilton T, Carr M, Harvey M. Virtual reality videos used in undergraduate palliative and oncology medical teaching: results of a pilot study. BMJ Supportive & Palliative Care. 2019; 9(3):281-5.
34. Anbro SJ, Szarko AJ, Houmanfar RA, Maraccini AM, Crosswell LH, Harris FC, et al. Using Virtual Simulations to Assess Situational Awareness and Communication in Medical and Nursing Education: A Technical Feasibility Study. Journal of Organizational Behavior Management. 2020; 40(1-2):129-39.
35. Schulz KF, Grimes DA. Unequal group sizes in randomised trials: guarding against guessing. Lancet. 2002; 359(9310):966-70.
36. Chirico A, Cipresso P, Yaden DB, Biassoni F, Riva G, Gaggioli A. Effectiveness of Immersive Videos in Inducing Awe: An Experimental Study. Sci Rep. 2017; 7(1):1218.
37. Tang KS, Cheng DL, Mi E, Greenberg PB. Augmented reality in medical education: a systematic review. Canadian medical education journal. 2020; 11(1):e81-e96.
38. Learning from Computer Games: A Research Review. Serious Games: The Challenge; 2012. Springer Berlin Heidelberg: Berlin, Heidelberg; 2012.