Introduction

Every day, millions of people worldwide refer to the community pharmacies for their health care needs. There is a growing global trend for patients to demand over-the-counter (OTC) drugs for treatment of their symptoms before going to the physician ( 1 ). Non-prescription therapy is a health-related topic that is being debated in most nations right now ( 2 ). Because community pharmacists are the most accessible healthcare experts, their role as counselor and prescribers of non-prescription drugs has become critical ( 3 , 4 ). Pharmacists are responsible to identify the patient symptoms and give proper advice ( 5 ). Nowadays, the pharmacist’s role is changing from drug dispensing to providing drug information and patient care ( 6 ). In Iran, the role of community pharmacists is not clearly described. In fact, most of the pharmacists are limited to being involved in prescription filling. Although the number of non-prescription drugs in Iran is limited, their consumption is overused ( 2 ). For example, since prescribing any type of OTC antibiotics is prohibited according to Iranian regulations, antibiotics were administered without any prescriptions, and in more than 90% of cases, the estimated time for each case was less than 60 seconds by the pharmacist ( 7 ). As usual, obtaining a patient's history in pharmacy is incomplete and unclassified ( 5 ), and the medication is given to the patient without the necessary recommendations in which the lack of professional knowledge and skills among pharmacists is one of the reasons to provide inadequate counseling to patients ( 8 , 9 ).

Nowadays, mobile health (mHealth) technology has been defined by the World Health Organization (WHO) for medical and public health practice ( 10 ). Applications used in health care settings have several functions, such as health record maintenance and monitoring, information gathering, and clinical decision making ( 11 ). Research showed that pharmacists were enthusiastic about using mobile devices as clinical references (e.g., UpToDate) to increase the patient care and medication safety ( 12 ). However, few mobile applications were designed to complement the pharmaceutical skills of pharmacists ( 13 ). By searching mHealth applications, we identified the most popular online markets such as iTunes ( 14 ), Google Play ( 15 ) and BlackBerry ( 16 ) via the “Medical” application categories ( 17 ) and keyword searches (e.g., non-prescription medicines, over the counter, OTC therapy, pharmacists) and search on databases (PubMed, ScienceDirect, MEDLINE, Scopus, Google Scholar, Elsevier, Web of science); this was conducted on December 2021, but there was no application among them based on Clinical Decision Support Systems (CDSS) to help the pharmacists in OTC management in pharmacies.

CDSS is designed to provide the users to support clinical decision-making and manage the patient’s condition ( 18 ). According to these problems, creating a user-friendly application that could help pharmacists obtain a complete history, determine referrals, and provide appropriate pharmacological and non-pharmacological recommendations is required. This study aimed to design, develop, and evaluate the effectiveness of OTC therapy application to achieve the practice change and examine the quality of application among a population of Iranian pharmacists who work at the pharmacies of Tehran University of Medical Science (TUMS) to elucidate their knowledge toward non-prescription medicines in the community pharmacists.

Methods

This study consisted of two phases, with the first phase involving the design and development of the application, and the second phase assessing the effectiveness of the application in bringing about changes in professional behavior before and after conducting this quasi-experimental study and evaluating the quality of the OTC therapy application among community pharmacists affiliated with TUMS in Tehran, Iran. The study lasted fifteen months from October 2020 to January 2022. Ethics committee of TUMS approved the protocol of the study (IR.TUMS.TIPS.REC.1400.038). Pharmacists were assured that participation in the study and use of the application were optional. Moreover, using the application for pharmacists was free. All information of the pharmacists were kept confidential, and the points obtained by pharmacists did not have any negative consequences on their jobs and financial situation.

The study included two sampling sections. The first section of sampling was done by convenience method to assess the effectiveness of the application before and after the quasi-experimental study. The second section of sampling was performed to evaluate the quality of OTC therapy application.

In addition to the data collected through the study, other data were collected electronically via an application consisting of two parts: i) demographic data and professional information of pharmacists and ii) user version of mobile application rating scale (uMARS) questionnaire ( 19 ).

A. Design and Development of the Application (First Phase)

A web-based needs assessment (https://epoll.pro/) regarding the most popular topics of OTC therapy was performed. This survey was conducted on the pharmacists, and then ten most popular topics were selected. Acne, obesity, ocular problems (eye redness and eyelid issues), sleep disorders, dermatitis, dysmenorrhea, hair loss, headache, burning, and vaginal infections were discussed. The OTC therapy of these topics was derived according to the latest version of OTC therapy textbooks ( 20 - 25 ). The OTC drugs list was extracted, and the Food and Drug Administration of Iran website (Jun 2021) and appropriate algorithms for each topic in the form of a decision tree were drawn. The scientific content of the algorithms was designed and categorized based on taking the patients’ history (patients’ demographic information, comorbid symptoms, time of the onset of symptoms, severity of symptoms, what medication improved or did not improve the condition, medical history of the patient, history of medications used, family history, food or drug allergy, specific population (pregnancy, lactation, children, the elderly, liver failure, renal failure)), and correct referral of patients to the physician and providing pharmacological and non-pharmacological recommendations. Two other clinical pharmacists evaluated and modified the formal and content validity of the algorithms initially. The final version of the algorithms was then authorized by four experienced clinical pharmacists and one specialist in information technology. Then, based on the designed algorithms, the application was developed using Asp.net with a Microsoft SQL Server database platform (http://sickness.online-demo.ir/Login.aspx). The application could be used online via personal computer (PC) or mobile (Android or iOS). The application was designed based on Mobile CDSS to give productive administration of information ( 26 ). The quality of OTC therapy application was evaluated and approved by five informatics specialists through a mobile application rating scale (MARS) questionnaire ( 27 ). After applying the changes, the final version of the application was approved by an expert panel consisting of four clinical pharmacy specialists and one informatics expert.

B. Evaluation of the Effectiveness of the Application (Second Phase)

The effectiveness of this application was evaluated via the before and after quasi-experimental study. Therefore, based on the method of sampling (convenience method) among the pharmacists working in the educational pharmacies of TUMS (N=120), only 60 pharmacists had the inclusion criteria of this study. Finally, among 60 pharmacists, only 40 agreed to participate in the study (Acceptance rate = 66.66%).

Based on the algorithms and application items, related scenarios and checklists were created for the ten topics listed above. They were verified by a panel of experts. Each checklist was constructed, so that the first section contained "taking a medical history" and the second section included patient treatment by "referring the patient to a physician" or offering "pharmacological and non-pharmacological advice." All the designed scenarios continued to the end of the path and did not immediately end in "referral to physicians".

The participants had to manage the scenarios first based on their knowledge, and then use the application. Before starting the study, the pharmacists were completely informed about all aspects of the study. We have covered 10 topics for OTC therapy, and according to a randomly prepared list (by a statistic specialist), the topics were numbered one to ten and divided among 40 pharmacists. Each topic was repeated 4 times (each pharmacist reviewed only one scenario). They randomly chose the subjects’ diseases scenarios (they were totally blinded about the scenarios’ topic).

Two clinical pharmacists who were aware of Iran OTC drugs list performed the study evaluation. There was a specific checklist for every scenario which was used for scoring each evaluated pharmacist. The scenario scoring method was as follows: score "1" or score "zero" was given if the question or point that the pharmacists expressed about the scenario was correct or incorrect, respectively. Score "-1" would be assigned if the recommended medicine by pharmacists was not included in OTC therapy list of Iran. The first evaluator (clinical pharmacist) appeared in the patient role and asked the pharmacist to manage the chief complaint of the relevant scenario. After the pharmacists listened to the case and collected the patient's medical history, they made clinical suggestions based on their expertise and without using the application. At the same time, another evaluator scored the pharmacist's answers based on each scenarios’ related checklist. The scores of “taking medical history” and providing “pharmacological and non-pharmacological recommendations” sections were recorded separately. To avoid bias, both clinical pharmacists were regularly present for all participants.

In the next step, the pharmacists were asked to manage the same scenarios using the application. Therefore, the application setting was described for the pharmacist. After getting acquainted with the application setting, each pharmacist created an account for himself/herself, and each person provided his/her own demographic information (age, gender) and professional information (university of study, year of graduation, work experience in pharmacy (year) and history of post-graduation training OTC therapy) and registered them in the user account. Then, the previous scenario was repeated, and users responded using the application. All conversations and time of case management before and after using the application separately were recorded to ensure proper evaluation. The scores were re-checked with the recorded voice.

C. Assessment the Quality of the Application (Second Phase)

The sample size was calculated to evaluate the precision, considering the accuracy rate equal to 80% for the confidence level of 95% and the estimation error of 15% value of 28. According to the study carried out by Alroobaea et al., for evaluating the quality of application, at least 97% of the defects could be identified by having the opinions of 30 users ( 28 ). Thus, the same 40 pharmacies that participated in before and after quasi-experimental study were asked to check all 10 topics in the application over a 30-day period. Referral to the application occurred at least 400 times by 40 users.

To evaluate the quality of the application, the survey was carried out using the uMARS questionnaire which was translated into Persian. The validity of the questionnaire was evaluated, and the reliability of the questionnaire was estimated using Cronbach's alpha=0.938 ( 19 , 29 ). The users were asked to rate the application after checking all 10 topics using the uMARS questionnaire. Each user, only after assessing ten topics of application, was allowed to access uMARS questionnaire, and there was only one time for the users to score the uMARS questionnaire.

The uMARS has 23 items which are categorized in 5 sections including: engagement (5 items), functionality (4 items), aesthetics (3 items), information quality (7 items), and subjective quality scale (4 items). Scores of all items have a maximal possible value of 5. Each section (A-B-C-D-E) is scored on a 5-point scale from 1 (“inadequate”) to 5 (“excellent”), with more specific descriptors for the response options for each question. The distinct component of this questionnaire (F section), which consists of six questions, assesses the perceived influence of an application on the user's awareness, knowledge, orientation, attitudes, and intentions to alter the user's mode of operation, and aids the user in searching. Upon completion, seven scores are reported, including the mean scores for each subscale (A, B, C, D sections), a total mean score, a mean subjective quality score (E section) and a mean score of app-specific subscale, perceived impact section of application (F section). All items of F section are ranked 1 (“strongly disagree”) to 5 (“strongly agree”) accordingly ( 30 ).

Statistical Analysis

The mean and standard deviation (SD) were calculated for age, and median (interquartile range (IQR)) described the work experience (year). For the categorical variables, gender and university of study, frequency, and percentage were calculated. To compare before/after measurements of parametric and non-parametric data, we used the paired t-test and Wilcoxon matched-pairs test, respectively. Besides, the variables were compared using Mann-Whitney test.

The median scores were obtained (N=4 pharmacist) and the times recorded for each scenario management, before and after using the application, were compared using Wilcoxon matched-pairs test.

The obtained mean scores and the times recorded for all 10 topics by 40 pharmacists before and after using the application were compared using paired t-test and Wilcoxon sign-rank test, respectively. The score of each pharmacist was divided by the maximum score that could be obtained in the relevant scenario. Besides, for scaling the time, each pharmacist's recorded time was divided by the maximum recorded time. The scaled numbers were calculated between 0 and 1. Then, regardless of the scenarios' subject, the scores were compared. Then, the scores of different parts of the checklists (“taking medical history”, “pharmacological and non-pharmacological recommendations”) were scaled, and the median scores were compared before and after using the application with Wilcoxon signed-rank test.

The correlation among the years of experience and the mean scaled scores before and after using the application was measured using the Spearman test. The relationship between gender and the mean scaled scores was evaluated using Mann-Whitney test. Mann-Whitney test assessed the relationship between gender and the median scores of different sections of the uMARS questionnaire. Furthermore, the correlation of years of experience in pharmacy with the median scores of different sections of the uMARS questionnaire was analyzed using the Spearman test. The statistical significance was considered at P<0.05. The analyses were performed using the statistical software Stata (ver. 13).

Results

Forty pharmacists participated in this before-after quasi-experimental study cross-sectional study. The sample was composed of thirty-one (77.5%) females and nine (22.5%) males. Thirty-three (82.5%) of the pharmacists graduated from TUMS. The pharmacy practice experience of the participants was from 0 to 27 years, with a median of 5 years. 47.5% (N=19) of the participants worked less than 5 years. None of the pharmacists had a history of post-graduation training OTC therapy.

At the first, data analysis of the obtained scores and the times recorded were performed for each scenario (subject-oriented) among 4 participants. All scores after using the application increased; there was no significant difference in the score of pharmacists before and after using the application (P-value = 0.06) (Table 1). The maximum improvement scores after using the application were obtained in vaginal infections. Although the recorded time was increased after the use of the application, the difference was not significant (P-value = 0.06) (Table 2).

The mean scaled scores of pharmacists before and after the use of the application were 0.27±0.15 and 0.72±0.20, respectively (P-value <0.001). The median scaled times before and after the use of application were equal to 150.5 second (101-203.5) and 411 second (343.5-515.5), respectively (P-value< 0.001). The scaled scores in different sections of the checklist among 40 pharmacists before and after the use of the application were compared. Median scaled scores after using the application in “taking medical history” section changed from 0.3 to 0.8, and in “pharmacological and non-pharmacological recommendations” section increased significantly from 0.11 to 0.76 (P-value <0.001). Regarding none of the scenarios requiring referring, the referral percentage was reduced from 32.5% to 2.5% after using the application.

There was no significant correlation between the years of experience and scaled scores before (P-value=0.65, r=0.07) and after the use of the application (P-value=0.50, r=0.10). The mean scaled scores of pharmacists before the use of the application by was 0.26±0.17 in those with working years less than 5, and 0.27±0.14 (P-value=0.72) in those whose working years were more than 5. However, after using the application, the mean scaled scores of pharmacists was 0.69±0.23 in those whose years of experience were less than 5 and 0.74±0.17 (P-value=0.58) in the pharmacists with an experience of more than 5.

The relationship between gender and the mean scaled scores before and after using the application was investigated. The results showed that there was no significant difference between gender and scaled scores of before and after using the application (P-value was 0.20 and 0.72, respectively).The result of uMARS questionnaire for 10 scenarios which were evaluated by 40 pharmacists (400 visits) is presented in Table 3. Mean scores of uMARS questionnaire sections had a minimum of 3. The highest mean scores of different uMARS questionnaire sections were related to the “Information” section (4.06±0.72) and “App perceived impact” section (3.57±0.74). In part E, one of the subscales, which was about paying for an application, 9 (22.5%) users were willing to pay, 14 (35%) were not willing to pay, and 17 (42.5%) users might pay.

In Table 4, the relationship between gender and the median scores of various parts of the uMARS questionnaire was evaluated. There was no relationship between gender and the median scores of various uMARS questionnaire parts. Moreover, the correlation of years of experience in pharmacy with the median scores of different sections of the uMARS questionnaire was investigated. There was only a significant correlation between years of experience and the median score of the App perceived impact (section F) in the uMARS questionnaire (P-value< 0.001 and r =-0.48).

Discussion

Given the importance of OTC medicines, a pharmacist is expected to have knowledge and experience on all OTC products to properly recommend to the drugs ( 31 ). As stated before, the application was created based on CDSS to assist the practitioners with decision-making and provide a proper approach to improve their performance and patient outcomes ( 32 , 33 ). Using the diagnostic decision support systems could save the times for practitioners ( 34 ). Since we do not have an application based on CDSS in OTC therapy, this application was designed, developed, and evaluated.

The results of our study showed that the time required to manage each scenario after using the OTC therapy application increased. There are several reasons for increasing the consumed time after using the application. It is possible that the pharmacists with OTC application spent more time to take the patient’s history (demographic information, age, self or someone else, drug history, family history, other signs and symptoms, onset and duration of chief complaints, severity of signs and symptoms, relieved or aggravated factors). They were more motivated to obtain a suitable consultation for patient evaluation as a result of the application's classified and coherent structure. Perhaps, the lack of experience in working with the application or a long time since the graduation of pharmacists is the reason for the slow use of the application. Another reason to increase the consultation time by application is that pharmacists without the application could not remember all the questions that should be asked during the consultation, so without application the consultation ended in a shorter time. Although there was no relationship between gender and the time consumed, the result of Jacobs’s study showed that being a female was associated with less technology readiness ( 35 ).

However, the main purpose of the study was improving the knowledge and skills of professional pharmacists in pharmacies. By comparing the data recorded in the scoring before and after the use of the application, a significant increase in the management of OTC counseling was shown in all the scenarios. This difference was not significant. This result might be explained by the small sample size. The data of the consumed times and obtained scores were scaled to increase the sample size and make them comparable. Significant differences were achieved after using the application. Comparing the years of experience (< 5 or ≥ 5 years) and mean scaled scores before and after using the application did not show an increase in the mean scaled scores. The application could change the professional behavior of the pharmacists because the scores of different parts of the checklists including “taking history” and “pharmacological and non-pharmacological recommendation” or “referring to a physician” among pharmacists regardless of the years of experience after the use of the application were increased. These differences were significant. This means that the application could improve the pharmacist’s management, so that they devoted more time to the patient for taking complete history and giving proper counseling. Moreover, after using the application, the percentage (30%) of referral to the physician wrongly decreased. On the other hand, the application helped the users to commensurate the patient's condition and refer or manage the patient.

The objective of Fernández-Lao et al.’s research was to assess the efficacy of a mobile learning (m-learning) program as a supplement to conventional education for the development of ultrasonography and palpation skills in the shoulder area among undergraduate physiotherapy students. The scores were significantly higher in the m-learning group than the traditional education group for the majority of items in the ultrasound assessment. As to our study, the time taken to get reliable skills in both groups was assessed. Students in the m-learning group spent more time situating the probe to collect an ultrasound picture of a particular structure because they were more driven to get a high score on the ultrasound evaluation ( 36 ). The systematic review conducted by Free et al. assessed the effectiveness of mobile technology interventions delivered to behavior change or disease management for health care consumers. Some trials indicated that receiving a mobile phone-based tele-monitoring intervention such as medication reminders, text messages, and video messages, significantly improved the quality of life and cure rates for diseases. However, some trials showed that, to date, m-health interventions for controlling diseases that have statistically significant effects are small and of borderline clinical importance. Nevertheless, evidence showed that short-term benefits for interventions could be clinically significant if sustained ( 37 ).

The research by Arnhold et al. investigated the usability of a commercially available diabetes application and concluded that its primary target users, senior diabetics in geriatrics, lacked usability ( 38 ). This conclusion underscores the need for evaluating the efficacy of applications, as the efficacy of OTC medication was investigated in our research since the evaluation of the performance of any information system is important for further improvement of that system ( 39 ). Unfortunately, many m-health applications are not evaluated ( 40 , 41 ), but similar studies have used uMARS questionnaire to evaluate the quality of the applications ( 34 , 42 ). The quality of the application was evaluated by uMARS questionnaire. The uMARS is a reliable, simple, multidimensional scale with validated scoring instrument of mHealth application quality, used to assess the quality of applications for management ( 42 ). The results of uMARS in this study in “Engagement”, “Functionality” and “Aesthetics” sections (Part A, B and C) were “Acceptable-Good” from the users' point of view. About “Information” section (Part D) “Good-Great” feedback was given. In general, “App quality score” section was reported to be “Acceptable-Good”. Also, “App subjective score” section (Part E) and “behavior change” section (Part F) were given values as respectively “Acceptable-Good” and “Undecided-Agree”.

As to the correlation Part F (App perceived impact) of uMARS questionnaire in OTC therapy application and years of experience, pharmacists with less experience in a pharmacy assigned more points to Part F. It means that this change was more bolded in pharmacists with less experience in pharmacy. From a different perspective, however, it is possible that more experienced (and older) pharmacists were unfamiliar with the application and did not answer the questionnaire correctly, whereas the younger pharmacists, who are part of the Internet generation, were able to better understand and use the application and, therefore, assigned a higher score to the questions. As Kenny et al. evaluated the adoption of m-health across generations, many older users do not have the ability to make practical and effective use of m-health applications, and older adults were slower than younger ones as to the use of the applications ( 43 ).

User suggestions in the uMARS questionnaire were to improve the application graphics and adding the other OTC therapy topics in the application. In general, in the future, efforts should be made to improve the weaknesses of the “Aesthetics” and “Engagement” by improving the graphical dimension of the application. When a user downloads an application, the developer has only one chance to get the user’s attention to use the application again. If the application does not meet the user’s needs, it will be the last time for the user to use the application. Therefore, with enhancements to graphic dimension, the users’ desire to use the program grows and the treatment of patients is enhanced (Part F score increases). In designing an application, it is crucial to examine and explain user requirements and make judicious decisions on the features to include.

Limitations

The lack of coverage of the entire community of pharmacists working in urban and non-university pharmacies (due to time, space, and financial constraints) was one of the limitations. Moreover, the sample size was small, so each of the ten topics were repeated four times (N=40).

Conclusions

The application developed in this study may be helpful to Persian-speaking pharmacists to increase their knowledge and pharmaceutical skills in OTC therapy in the mentioned 10 topics. OTC therapy application helps to categorize the information to guide the pharmacists in the above-mentioned situations. All parts of application were accepted by the users, but the graphics of application should be improved to be user-friendly. Considering the effect of OTC therapy application, there is a need for policymakers’ support to add other OTC topics in the application, improve its graphics, evaluate the application in a wider population, and distribute the application among all community pharmacists in the country.

Acknowledgments

This research was supported by Tehran University of Medical Sciences & Health Services grant. The authors are grateful for the collaboration of the pharmacists who participated in the study. Furthermore, the authors are thankful to Mr. Mohammad Zamani and his team for development of the application.

Authors’ Contributions

S.N, Sh.E, M.M; Investigation: S.N, Sh.E, M.M, H.ZH and P.P; Formal Analysis: S.N, P.P; Writing–Original Draft: P.P; Writing–Review &Editing: S.N, Sh.E, M.M and P.P; Visualization: S.N, Sh.E, M.M, H.ZH and P.P; Project Administration: S.N. All authors contributed to the discussion, read, and approved the manuscript and agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Conflict of interest

None Declared.

References

1. Moritz K, Seiberth JM, Herrmann NS, Bertsche T, Schiek S. Are evidence-based criteria addressed during counseling on over-the-counter products? An observational study in community pharmacies. Patient Education and Counseling. 2021; 104(11):2824-9.
2. Hanafi S, Poormalek F, Torkamandi H, Hajimiri M, Esmaeili M, Khooie SH, et al. Evaluation of community pharmacists’ knowledge, attitude and practice towards good pharmacy practice in Iran. Journal of pharmaceutical care. 2013; 1(1):19-24.
3. Chang J, Lizer A, Patel I, Bhatia D, Tan X, Balkrishnan R. Prescription to over-the-counter switches in the United States. Journal of Research in Pharmacy Practice. 2016; 5(3):149.
4. Tan BS, Goh AW, Chong CP. Utilization of non-prescription medicines in minor ailments management: A nationwide study undertaken among Malaysian community pharmacists. Journal of Applied Pharmaceutical Science. 2020; 10(3):59-66.
5. Negru DS, Cristea AN, Petculescu AM. Minimal consultation provided to patients presenting symptoms in the Romanian community pharmacy. Farmacia. 2010; 58:601-10.
6. Yeğenoğlu S, Çelikay G. Counselling of pharmacists to community on issues other than drug purchasing and drug related information: A survey in Ankara. Turkish Journal of Pharmaceutical Sciences. 2005; 2(2):83-91.
7. Soltani D, Ebrahimpour S, Hajimiri SH, Kohneloo AJ, Jahangard-Rafsanjani Z. Assessment of Antibiotic Dispensing Practice in Community Pharmacies of Tehran, for 2 Common Infectious Symptoms, Using Simulated Patient Method. Iranian journal of pharmaceutical research: IJPR. 2020; 19(4):35.
8. Zare M, Zarei L, Afifi S, Karimzadeh I, Ghaeminia M, Peiravian F, et al. The pharmacists’ patient care process in community pharmacy setting: evidence from Iran. Acta Bio Medica: Atenei Parmensis. 2020; 91(3):e2020059.
9. Negru DS, Cristea AN, Petculescu AM. Patient counselling at dispensing OTC medicines in the community pharmacy. Farmacia. 2012; 60(1):102-10.
10. World Health Organization [Internet]. Geneva: WHO Press. mHealth New horizons for health through mobile technologies. 2011 [Accessed 2 Jan 2017]. Available from: http://www.who.int/goe/publications/goe_mhealth_web.pdf.
11. Zhao J, Freeman B, Li M. Can mobile phone apps influence people’s health behavior change? An evidence review. Journal of medical Internet research. 2016; 18(11):e5692.
12. Aungst TD. Medical applications for pharmacists using mobile devices. Annals of Pharmacotherapy. 2013; 47(7-8):1088-95.
13. Rivera J, McPherson A, Hamilton J, Birken C, Coons M, Iyer S, et al. Mobile apps for weight management: a scoping review. JMIR mHealth and uHealth. 2016; 4(3):e5115.
14. iTunes Apple [Internet]; 2022 [Accessed 20 Jan 2022]. Available from: http://www.apple.com/itunes.
15. Android Apps on Google Play [Internet]; 2022 [Accessed 7 Feb 2022]. Available from: https://play.google.com/store/apps.
16. BlackBerry App World [Internet]; 2017 [Cited 25 April 2017]. Available from: https://appworld.blackberry.com.
17. Short SS, Lin AC, Merianos DJ, Burke RV, Upperman JS. Smartphones, trainees, and mobile education: implications for graduate medical education. Journal of graduate medical education. 2014; 6(2):199-202.
18. Seppälä J, De Vita I, Jämsä T, Miettunen J, Isohanni M, Rubinstein K, et al. Mobile phone and wearable sensor-based mHealth approaches for psychiatric disorders and symptoms: systematic review. JMIR mental health. 2019; 6(2):e9819.
19. Raeisi A. Review of AIDS mobile applications and assessment of their quality using the Mobile Application Rating Scale [Doctoral dissertation].. School of Health Information and Management: Kerman University of Medical Sciences; 2021.
20. NHS England. Self-care and over the counter items: A quick reference guide. Guidance for CCGs: UK; 2018.
21. Zgarrick DP, Alston GL, Moczygemba LR, Desselle SP. Pharmacy Management: Essentials for all practice settings. McGraw Hill Medical: USA; 2020.
22. Nathan A. Non-prescription medicines. Pharmaceutical press: London; 2010.
23. Rutter P. Community pharmacy: symptoms, diagnosis and treatment. Elsevier: UK; 2020.
24. Krinsky D, Ferreri S, Hemstreet B, Hume A, Newton G, Rollins C, et al. Handbook of Nonprescription Drugs: An Interactive Approach to Self-Care. American Pharmacists Association: USA; 2019.
25. Blenkinsopp A, Duerden M, Blenkinsopp J. Symptoms in the pharmacy: a guide to the management of common illnesses. John Wiley &amp; Sons: New York; 2020.
26. Kashyap R. Systematic Model for Decision Support System: In Interdisciplinary Approaches to Information Systems and Software Engineering. IGI Global: USA; 2019.
27. Lambrecht A, Vuillerme N, Raab C, Simon D, Messner EM, Hagen M, et al. Quality of a Supporting Mobile App for Rheumatic Patients: Patient-Based Assessment Using the User Version of the Mobile Application Scale (uMARS).. Front Med (Lausanne).. 2021; 8:715345.
28. Alroobaea R, Mayhew PJ. IEEE: USA.
29. Raeesi A, Khajouei R, Ahmadian L. Evaluation of HIV/AIDS-related mobile health applications content using an evidence-based content rating tool. BMC medical informatics and decision making. 2021; 21(1):1-2.
30. Roberts AE, Davenport TA, Wong T, Moon HW, Hickie IB, LaMonica HM. Evaluating the quality and safety of health-related apps and e-tools: Adapting the Mobile App Rating Scale and developing a quality assurance protocol. Internet Interventions. 2021; 24:100379.
31. Grebenar D, Nham E, Likic R. Factors influencing pharmacists’ over-the-counter drug recommendations. Postgraduate Medical Journal. 2020; 96(1133):144-8.
32. Haghighi PD, Burstein F, Churilov L, Patel A. Multi-Criteria Evaluation of Mobile Triage Decision Systems. InDSS 2014 May 22: India.
33. Berner ES, McGowan JJ. Use of diagnostic decision support systems in medical education. Methods of information in medicine. 2010; 49(4):412-7.
34. Knitza J, Tascilar K, Messner EM, Meyer M, Vossen D, Pulla A, et al. German mobile apps in rheumatology: review and analysis using the Mobile Application Rating Scale (MARS).. JMIR mHealth and uHealth. 2019; 7(8):e14991.
35. Jacobs RJ, Caballero J, Parmar J, Kane MN. The role of self-efficacy, flexibility, and gender in pharmacy students&#039; health information technology readiness. Currents in Pharmacy Teaching and Learning. 2019; 11(11):1103-10.
36. Fernández-Lao C, Cantarero-Villanueva I, Galiano-Castillo N, Caro-Morán E, Díaz-Rodríguez L, Arroyo-Morales M. The effectiveness of a mobile application for the development of palpation and ultrasound imaging skills to supplement the traditional learning of physiotherapy students. BMC medical education. 2016; 16(1):1-7.
37. Free C, Phillips G, Galli L, Watson L, Felix L, Edwards P, et al. The effectiveness of mobile-health technology-based health behaviour change or disease management interventions for health care consumers: a systematic review. PLoS medicine. 2013; 10(1):e1001362.
38. Arnhold M, Quade M, Kirch W. Mobile applications for diabetics: a systematic review and expert-based usability evaluation considering the special requirements of diabetes patients age 50 years or older. Journal of medical Internet research. 2014; 16(4):e2968.
39. Maynard S, Burstein F, Arnott D. A multi-faceted decision support system evaluation approach. Journal of decision systems. 2001; 10(3-4):395-428.
40. Payne HE, Lister C, West JH, Bernhardt JM. Behavioral functionality of mobile apps in health interventions: a systematic review of the literature. JMIR mHealth and uHealth. 2015; 3(1):e3335.
41. McKay FH, Cheng C, Wright A, Shill J, Stephens H, Uccellini M. Evaluating mobile phone applications for health behaviour change: A systematic review. Journal of telemedicine and telecare. 2018; 24(1):22-30.
42. Chavez S, Fedele D, Guo Y, Bernier A, Smith M, Warnick J, et al. Mobile apps for the management of diabetes. Diabetes Care. 2017; 40(10):e145-6.
43. Kenny G, Connolly R.Paper presented at: ;