Evaluation Methods in Clinical Health Technologies: A Systematic Review
Abstract
Background: This study was conducted to classify the types of evaluation methods in clinical health technologies based on a systematic review method.
Methods: An electronic search was conducted in three scientific databases including Scopus, PubMed and ISI. The search strategy was performed in Jul to Nov 2021 and based on the three main concepts of "evaluation", "technology", "health. This search has been restricted to 10 years (2011-2021). Moreover, it only was limited to English and papers published in journals and conferences proceeding.
Results: Overall, 8149 references were identified for title and abstract screening. Full text screening was performed for 2674 articles, with 174 meeting the criteria for study inclusion.
Conclusion: Most of the technologies evaluated in these articles were associated with PC-based systems (N=107), and there have been fewer mobile apps (N=67). Most of used technologies were with goals of treatment (43%, N=74) and education (26%, N=45). Among all the methods, the most and the least used methods were usability (66%, N=115) and qualitative (1%, N=2) method, respectively. The most method for health clinical technologies is usability method especially in telemedicine field.
1. Haried P, Claybaugh C, Dai H (2019). Eval-uation of health information systems re-search in information systems research: A meta-analysis. Health Informatics J, 25(1): 186–202.
2. Melia R, Francis K, Hickey E, et al (2020). Mobile health technology interventions for suicide prevention: systematic review. JMIR Mhealth Uhealth, 8(1): e12516.
3. Sheikh A, Sood HS, Bates DW (2015). Lev-eraging health information technology to achieve the “triple aim” of healthcare re-form. J Am Med Inform Assoc, 22(4): 849–56.
4. Nabovati E, Vakili-Arki H, Eslami S, Kha-jouei R (2014). Usability evaluation of la-boratory and radiology information sys-tems integrated into a hospital infor-mation system. J Med Syst, 38(4):35.
5. Donaldson MS, Corrigan JM, Kohn LT (2000). To err is human: building a safer health system. Vol. 6. National Academies Press. 376-379.
6. Skiba D (2017). Evaluation Tools to Ap-praise Social Media and Mobile Applica-tions. Informatics, 4(3): 32.
7. van der Vaart R, Drossaert C (2017). Devel-opment of the digital health literacy in-strument: measuring a broad spectrum of health 1.0 and health 2.0 skills. J Med In-ternet Res, 19(1): e27.
8. Fleming GA, Petrie JR, Bergenstal RM, et al (2020). Diabetes Digital App Technolo-gy : Benefits, Challenges, and Recom-mendations. A Consensus Report by the European Association for the Study of Diabetes (EASD) and the American Dia-betes Association (ADA) Diabetes Tech-nology Working Group. Diabetes Care, 43(1):250-260.
9. HoltzB, Vasold K, Cotten S, et al (2019). Health care provider perceptions of con-sumer-grade devices and apps for track-ing health: a pilot study. JMIR Mhealth Uhealth, 7(1):e9929.
10. Marzano L, Bardill A, Fields B, et al (2015). The application of mHealth to mental health: opportunities and challenges. Lan-cet Psychiatry, 2(10): 942–8.
11. Burke LE, Warziski M, Starrett T, et al (2005). Self-monitoring dietary intake: current and future practices. J Ren Nutr, 15(3): 281–90.
12. McClellan SR, Casalino LP, Shortell SM, et al (2013). When does adoption of health in-formation technology by physician prac-tices lead to use by physicians within the practice? J Am Med Inform Assoc, 20(e1): e26–32.
13. Zettel-Watson L, Tsukerman D (2016). Adoption of online health management tools among healthy older adults: An ex-ploratory study. Health Informatics J, 22(2): 171–83.
14. Pravettoni G, Triberti S (2020). A ``P5’’ Ap-proach to Healthcare and Health Technology. In: Pravettoni G, Triberti S, editors. P5 eHealth: An Agenda for the Health Technologies of the Future Springer, Cham. https://doi.org/10.1007/978-3-030-27994-3_1
15. Chang YS, Zhang Y, Gwizdka J (2021). The effects of information source and eHealth literacy on consumer health in-formation credibility evaluation behavior. Comput Human Behav, 115: 106629.
16. Velickovski F, Ceccaroni L, Roca J, et al (2014). Clinical Decision Support Systems (CDSS) for preventive management of COPD patients. J Transl Med, 12 Suppl 2(Suppl 2):S9.
17. McAlearney AS, Hefner JL, MacEwan SR, et al (2021). Care Team Perspectives about an Inpatient Portal: Benefits and Chal-lenges of Patients’ Portal Use During Hospitalization. Med Care Res Rev, 78(5): 537-547.
18. Jordans M, Chisholm D, Semrau M, et al (2019). Evaluation of performance and perceived utility of mental healthcare indi-cators in routine health information sys-tems in five low-and middle-income countries. BJPsych Open, 5(5): e70.
19. Safdari R, Shams Abadi A. R, & Pahlevany Nejad S (2018). Improve health of the el-derly people with M-health and technol-ogy. Iran J Ageing, 13(3): 288-299.
20. Blandford A, Gibbs J, Newhouse N, et al (2018). Seven lessons for interdisciplinary research on interactive digital health inter-ventions. Digital Health, 4. doi:10.1177/2055207618770325
21. Murray E, Hekler EB, Andersson G, et al (2016). Evaluating Digital Health Inter-ventions: Key Questions and Approach-es. Am J Prev Med, 51(5): 843–51.
22. Zhao J, Ma Z, Luo Z, Li Z, Zhi Y (2019). The Necessity of Information Communication Equipment Status Evaluation. In 3rd Interna-tional Conference on Mechatronics Engineering and Information Technology (ICMEIT 2019). Atlantis Press: pp. 84-88.
23. Buntin MB, Burke MF, Hoaglin MC, Blu-menthal D (2011). The benefits of health information technology: A review of the recent literature shows predominantly positive results. Health Aff (Millwood), 30(3): 464–71.
24. Chaudhry B, Wang J, Wu S, et al (2006). Sys-tematic review: impact of health infor-mation technology on quality, efficiency, and costs of medical care. Ann Intern Med, 144(10): 742-752.
25. Smith A “Ben,” Bamgboje‐Ayodele A, Bu-tow P, et al (2020). Development and us-ability evaluation of an online self‐management intervention for fear of can-cer recurrence (iConquerFear). Psychooncolo-gy, 29(1): 98–106.
26. Mohammadzadeh N, Khenarinezhad S, Gha-Zanfarisavadkoohi E, et al (2021). Evaluation of m-health applications use in epilepsy: A systematic review. Iran J Public Health, 50(3): 459–469.
27. Sligo J, Gauld R, Roberts V, Villa L (2017). A literature review for large-scale health information system project planning, im-plementation and evaluation. Int J Med In-form, 97: 86–97.
28. Greenhalgh T, Potts HWW, Wong G, et al (2009). Tensions and paradoxes in elec-tronic patient record research: A system-atic literature review using the meta-narrative method. Milbank Q, 87(4): 729–88.
29. Cook TD, Reichardt CS (1979). Qualitative and quantitative methods in evaluation research. Vol. 1. Sage publications Beverly Hills, CA.
30. Heeks R (2006). Health information systems: Failure, success and improvisation. Int J Med Inform, 75(2): 125–37.
31. Oates BJ, Wainwright DW, Edwards HM (2013). Endless Bad Projects or Evidence-Based Practice? An Agenda for Action. In: Interna-tional Working Conference on Transfer and Dif-fusion of IT. Springer, 619–624.
32. Dujmovic JJ (2008). Continuous Preference Logic for System Evaluation. IEEE Trans Fuzzy Syst, 15(6): 1082–1099.
33. Fridrich A, Jenny GJ, Bauer GF (2015). The context, process, and outcome evaluation model for organisational health interven-tions. Biomed Res Int, 2015:414832.
34. Cronholm S, & Goldkuhl G (2003). Strate-gies for information systems evaluation-six generic types. Electronic Journal of Infor-mation Systems Evaluation, 6(2): 65-74.
35. Irani Z (2002). Information systems evalua-tion: Navigating through the problem domain. Information & Management, 40: 11–24.
36. Shaout A, Yousif MK (2014). Performance Evaluation – Methods and Techniques Survey. International Journal of Computer and Information Technology, 03(05): 966-979.
37. Chen S, Osman NM, Nunes MB, Peng GC (2011). Information systems evaluation methodologies. Proceedings of the IADIS International Workshop Infor-mation Systems Research Trends, Ap-proaches and Methodologies, ISRTAM 2011, Part of the IADIS, MCCSIS 2011: 23–32.
38. Khajouei R, Jaspers MWM (2010). The im-pact of CPOE medication systems’ de-sign aspects on usability, workflow and medication orders. Methods Inf Med, 49(1): 3–19.
39. Wiegel V, King A, Mozaffar H, et al (2020). A systematic analysis of the optimization of computerized physician order entry and clinical decision support systems: a qualitative study in English hospitals. Health Informatics J, 26(2):1118–32.
40. Kuotu G. C, & Moukam Lower B. B (2020). Literature study on the return on invest-ment concerning the implementation of a computerized clinical decision support in a hospital information system. medRxiv, https://doi.org/10.1101/2020.10.30.20223362
41. Highfill T (2020). Do hospitals with elec-tronic health records have lower costs? A systematic review and meta-analysis. Int J Healthc Manag, 13(1):65-71.
42. Davis R, Gardner J, & Schnall R (2020). A review of usability evaluation methods and their use for testing eHealth HIV in-terventions. Curr HIV/AIDS Rep, 17(3): 203-218.
43. Dhouib A, Trabelsi A, Kolski C, Neji M (2016). A classification and comparison of usability evaluation methods for inter-active adaptive systems. In: Proceedings - 2016 9th International Conference on Human System Interactions, HIS, 2016: 246–51.
44. Bhutkar G, Konkani A, Katre D, Ray GG (2013). A review healthcare usability eval-uation methods. Biomed Instrum Technol, Suppl:45-53.
45. Ellsworth MA, Dziadzko M, O’Horo JC, et al (2017). An appraisal of published usa-bility evaluations of electronic health rec-ords via systematic review. J Am Med In-form Assoc, 24(1): 218–26.
46. Bevan N, Carter J, Harker S (2015). ISO 9241-11 revised: What have we learnt about us-ability since 1998? In: International Conference on Human-Computer Interaction. Springer: 143–51.
47. Yun SJ, Kang M-G, Yang D, et al (2020). Cognitive Training Using Fully Immer-sive, Enriched Environment Virtual Reali-ty for Patients With Mild Cognitive Im-pairment and Mild Dementia: Feasibility and Usability Study. JMIR Serious Games, 8(4): e18127.
48. Thyvalikakath TP, Monaco V, Thambugani-palle HB, Schleyer T (2008). A usability evaluation of four commercial dental computer-based patient record systems. J Am Dent Assoc, 139(12): 1632–42.
49. Jaspers MWM (2009). A comparison of us-ability methods for testing interactive health technologies: methodological as-pects and empirical evidence. Int J Med In-form, 78(5): 340–53.
50. Khowaja K, Al-Thani D (2020). New Checklist for the Heuristic Evaluation of mHealth Apps (HE4EH): Development and Usability Study. JMIR Mhealth Uhealth. 8(10): e20353.
51. de Matos Lima S, Gimenez CL, Luna D (2020). Challenges of In-House Devel-opment and Implementation of a CPOE for Oncology. Stud Health Technol Inform, 270:1217-1218.
52. Classen DC, Holmgren AJ, Newmark LP, et al (2020). National trends in the safety performance of electronic health record systems from 2009 to 2018. JAMA Netw Open, 3(5):e205547.
53. Hersh WR, Helfand M, Wallace J, et al (2001). Clinical outcomes resulting from telemedicine interventions: a systematic review. BMC Med Inform Decis Mak, 1:5.
54. Godersky ME, Klein JW, Merrill JO, et al (2020). Acceptability and feasibility of a mobile health application for video di-rectly observed therapy of buprenorphine for opioid use disorders in an office-based setting. J Addict Med, 14(4): 319–25.
55. Bellei EA, Biduski D, Lisboa HRK, De Marchi ACB (2020). Development and assessment of a mobile health application for monitoring the linkage among treat-ment factors of Type 1 Diabetes Mellitus. Telemed J E Health, 26(2): 205–17.
56. Shah SGS, Robinson I (2008). Medical de-vice technologies: who is the user? Int J Healthc Technol Manag, 9(2): 181–97.
57. Bhutkar G, Katre D, Rajhans N, et al (2010). Analysis and design of ICU Knowledge Management System (IKMS) for Indian environment with usability perspective. In: 2010. The 2nd International Confer-ence on Computer and Automation En-gineering (ICCAE). IEEE: 329–33.
2. Melia R, Francis K, Hickey E, et al (2020). Mobile health technology interventions for suicide prevention: systematic review. JMIR Mhealth Uhealth, 8(1): e12516.
3. Sheikh A, Sood HS, Bates DW (2015). Lev-eraging health information technology to achieve the “triple aim” of healthcare re-form. J Am Med Inform Assoc, 22(4): 849–56.
4. Nabovati E, Vakili-Arki H, Eslami S, Kha-jouei R (2014). Usability evaluation of la-boratory and radiology information sys-tems integrated into a hospital infor-mation system. J Med Syst, 38(4):35.
5. Donaldson MS, Corrigan JM, Kohn LT (2000). To err is human: building a safer health system. Vol. 6. National Academies Press. 376-379.
6. Skiba D (2017). Evaluation Tools to Ap-praise Social Media and Mobile Applica-tions. Informatics, 4(3): 32.
7. van der Vaart R, Drossaert C (2017). Devel-opment of the digital health literacy in-strument: measuring a broad spectrum of health 1.0 and health 2.0 skills. J Med In-ternet Res, 19(1): e27.
8. Fleming GA, Petrie JR, Bergenstal RM, et al (2020). Diabetes Digital App Technolo-gy : Benefits, Challenges, and Recom-mendations. A Consensus Report by the European Association for the Study of Diabetes (EASD) and the American Dia-betes Association (ADA) Diabetes Tech-nology Working Group. Diabetes Care, 43(1):250-260.
9. HoltzB, Vasold K, Cotten S, et al (2019). Health care provider perceptions of con-sumer-grade devices and apps for track-ing health: a pilot study. JMIR Mhealth Uhealth, 7(1):e9929.
10. Marzano L, Bardill A, Fields B, et al (2015). The application of mHealth to mental health: opportunities and challenges. Lan-cet Psychiatry, 2(10): 942–8.
11. Burke LE, Warziski M, Starrett T, et al (2005). Self-monitoring dietary intake: current and future practices. J Ren Nutr, 15(3): 281–90.
12. McClellan SR, Casalino LP, Shortell SM, et al (2013). When does adoption of health in-formation technology by physician prac-tices lead to use by physicians within the practice? J Am Med Inform Assoc, 20(e1): e26–32.
13. Zettel-Watson L, Tsukerman D (2016). Adoption of online health management tools among healthy older adults: An ex-ploratory study. Health Informatics J, 22(2): 171–83.
14. Pravettoni G, Triberti S (2020). A ``P5’’ Ap-proach to Healthcare and Health Technology. In: Pravettoni G, Triberti S, editors. P5 eHealth: An Agenda for the Health Technologies of the Future Springer, Cham. https://doi.org/10.1007/978-3-030-27994-3_1
15. Chang YS, Zhang Y, Gwizdka J (2021). The effects of information source and eHealth literacy on consumer health in-formation credibility evaluation behavior. Comput Human Behav, 115: 106629.
16. Velickovski F, Ceccaroni L, Roca J, et al (2014). Clinical Decision Support Systems (CDSS) for preventive management of COPD patients. J Transl Med, 12 Suppl 2(Suppl 2):S9.
17. McAlearney AS, Hefner JL, MacEwan SR, et al (2021). Care Team Perspectives about an Inpatient Portal: Benefits and Chal-lenges of Patients’ Portal Use During Hospitalization. Med Care Res Rev, 78(5): 537-547.
18. Jordans M, Chisholm D, Semrau M, et al (2019). Evaluation of performance and perceived utility of mental healthcare indi-cators in routine health information sys-tems in five low-and middle-income countries. BJPsych Open, 5(5): e70.
19. Safdari R, Shams Abadi A. R, & Pahlevany Nejad S (2018). Improve health of the el-derly people with M-health and technol-ogy. Iran J Ageing, 13(3): 288-299.
20. Blandford A, Gibbs J, Newhouse N, et al (2018). Seven lessons for interdisciplinary research on interactive digital health inter-ventions. Digital Health, 4. doi:10.1177/2055207618770325
21. Murray E, Hekler EB, Andersson G, et al (2016). Evaluating Digital Health Inter-ventions: Key Questions and Approach-es. Am J Prev Med, 51(5): 843–51.
22. Zhao J, Ma Z, Luo Z, Li Z, Zhi Y (2019). The Necessity of Information Communication Equipment Status Evaluation. In 3rd Interna-tional Conference on Mechatronics Engineering and Information Technology (ICMEIT 2019). Atlantis Press: pp. 84-88.
23. Buntin MB, Burke MF, Hoaglin MC, Blu-menthal D (2011). The benefits of health information technology: A review of the recent literature shows predominantly positive results. Health Aff (Millwood), 30(3): 464–71.
24. Chaudhry B, Wang J, Wu S, et al (2006). Sys-tematic review: impact of health infor-mation technology on quality, efficiency, and costs of medical care. Ann Intern Med, 144(10): 742-752.
25. Smith A “Ben,” Bamgboje‐Ayodele A, Bu-tow P, et al (2020). Development and us-ability evaluation of an online self‐management intervention for fear of can-cer recurrence (iConquerFear). Psychooncolo-gy, 29(1): 98–106.
26. Mohammadzadeh N, Khenarinezhad S, Gha-Zanfarisavadkoohi E, et al (2021). Evaluation of m-health applications use in epilepsy: A systematic review. Iran J Public Health, 50(3): 459–469.
27. Sligo J, Gauld R, Roberts V, Villa L (2017). A literature review for large-scale health information system project planning, im-plementation and evaluation. Int J Med In-form, 97: 86–97.
28. Greenhalgh T, Potts HWW, Wong G, et al (2009). Tensions and paradoxes in elec-tronic patient record research: A system-atic literature review using the meta-narrative method. Milbank Q, 87(4): 729–88.
29. Cook TD, Reichardt CS (1979). Qualitative and quantitative methods in evaluation research. Vol. 1. Sage publications Beverly Hills, CA.
30. Heeks R (2006). Health information systems: Failure, success and improvisation. Int J Med Inform, 75(2): 125–37.
31. Oates BJ, Wainwright DW, Edwards HM (2013). Endless Bad Projects or Evidence-Based Practice? An Agenda for Action. In: Interna-tional Working Conference on Transfer and Dif-fusion of IT. Springer, 619–624.
32. Dujmovic JJ (2008). Continuous Preference Logic for System Evaluation. IEEE Trans Fuzzy Syst, 15(6): 1082–1099.
33. Fridrich A, Jenny GJ, Bauer GF (2015). The context, process, and outcome evaluation model for organisational health interven-tions. Biomed Res Int, 2015:414832.
34. Cronholm S, & Goldkuhl G (2003). Strate-gies for information systems evaluation-six generic types. Electronic Journal of Infor-mation Systems Evaluation, 6(2): 65-74.
35. Irani Z (2002). Information systems evalua-tion: Navigating through the problem domain. Information & Management, 40: 11–24.
36. Shaout A, Yousif MK (2014). Performance Evaluation – Methods and Techniques Survey. International Journal of Computer and Information Technology, 03(05): 966-979.
37. Chen S, Osman NM, Nunes MB, Peng GC (2011). Information systems evaluation methodologies. Proceedings of the IADIS International Workshop Infor-mation Systems Research Trends, Ap-proaches and Methodologies, ISRTAM 2011, Part of the IADIS, MCCSIS 2011: 23–32.
38. Khajouei R, Jaspers MWM (2010). The im-pact of CPOE medication systems’ de-sign aspects on usability, workflow and medication orders. Methods Inf Med, 49(1): 3–19.
39. Wiegel V, King A, Mozaffar H, et al (2020). A systematic analysis of the optimization of computerized physician order entry and clinical decision support systems: a qualitative study in English hospitals. Health Informatics J, 26(2):1118–32.
40. Kuotu G. C, & Moukam Lower B. B (2020). Literature study on the return on invest-ment concerning the implementation of a computerized clinical decision support in a hospital information system. medRxiv, https://doi.org/10.1101/2020.10.30.20223362
41. Highfill T (2020). Do hospitals with elec-tronic health records have lower costs? A systematic review and meta-analysis. Int J Healthc Manag, 13(1):65-71.
42. Davis R, Gardner J, & Schnall R (2020). A review of usability evaluation methods and their use for testing eHealth HIV in-terventions. Curr HIV/AIDS Rep, 17(3): 203-218.
43. Dhouib A, Trabelsi A, Kolski C, Neji M (2016). A classification and comparison of usability evaluation methods for inter-active adaptive systems. In: Proceedings - 2016 9th International Conference on Human System Interactions, HIS, 2016: 246–51.
44. Bhutkar G, Konkani A, Katre D, Ray GG (2013). A review healthcare usability eval-uation methods. Biomed Instrum Technol, Suppl:45-53.
45. Ellsworth MA, Dziadzko M, O’Horo JC, et al (2017). An appraisal of published usa-bility evaluations of electronic health rec-ords via systematic review. J Am Med In-form Assoc, 24(1): 218–26.
46. Bevan N, Carter J, Harker S (2015). ISO 9241-11 revised: What have we learnt about us-ability since 1998? In: International Conference on Human-Computer Interaction. Springer: 143–51.
47. Yun SJ, Kang M-G, Yang D, et al (2020). Cognitive Training Using Fully Immer-sive, Enriched Environment Virtual Reali-ty for Patients With Mild Cognitive Im-pairment and Mild Dementia: Feasibility and Usability Study. JMIR Serious Games, 8(4): e18127.
48. Thyvalikakath TP, Monaco V, Thambugani-palle HB, Schleyer T (2008). A usability evaluation of four commercial dental computer-based patient record systems. J Am Dent Assoc, 139(12): 1632–42.
49. Jaspers MWM (2009). A comparison of us-ability methods for testing interactive health technologies: methodological as-pects and empirical evidence. Int J Med In-form, 78(5): 340–53.
50. Khowaja K, Al-Thani D (2020). New Checklist for the Heuristic Evaluation of mHealth Apps (HE4EH): Development and Usability Study. JMIR Mhealth Uhealth. 8(10): e20353.
51. de Matos Lima S, Gimenez CL, Luna D (2020). Challenges of In-House Devel-opment and Implementation of a CPOE for Oncology. Stud Health Technol Inform, 270:1217-1218.
52. Classen DC, Holmgren AJ, Newmark LP, et al (2020). National trends in the safety performance of electronic health record systems from 2009 to 2018. JAMA Netw Open, 3(5):e205547.
53. Hersh WR, Helfand M, Wallace J, et al (2001). Clinical outcomes resulting from telemedicine interventions: a systematic review. BMC Med Inform Decis Mak, 1:5.
54. Godersky ME, Klein JW, Merrill JO, et al (2020). Acceptability and feasibility of a mobile health application for video di-rectly observed therapy of buprenorphine for opioid use disorders in an office-based setting. J Addict Med, 14(4): 319–25.
55. Bellei EA, Biduski D, Lisboa HRK, De Marchi ACB (2020). Development and assessment of a mobile health application for monitoring the linkage among treat-ment factors of Type 1 Diabetes Mellitus. Telemed J E Health, 26(2): 205–17.
56. Shah SGS, Robinson I (2008). Medical de-vice technologies: who is the user? Int J Healthc Technol Manag, 9(2): 181–97.
57. Bhutkar G, Katre D, Rajhans N, et al (2010). Analysis and design of ICU Knowledge Management System (IKMS) for Indian environment with usability perspective. In: 2010. The 2nd International Confer-ence on Computer and Automation En-gineering (ICCAE). IEEE: 329–33.
| Files | ||
| Issue | Vol 52 No 5 (2023) | |
| Section | Review Article(s) | |
| DOI | https://doi.org/10.18502/ijph.v52i5.12708 | |
| Keywords | ||
| Evaluation Health Technology Assessment Systematic review | ||
| Rights and permissions | |
|
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. |
How to Cite
1.
Mohammadzadeh N, Rahmani Katigari M, Hosseini R, PahlevanyNejad S. Evaluation Methods in Clinical Health Technologies: A Systematic Review. Iran J Public Health. 2023;52(5):913-923.
