Initial Development of User-Based Quality Evaluation Questionnaire of Smartwatch Technology for Applying to Healthcare
Abstract
Background: Smartwatches are a consumer wearable device offering a potential, practical, and affordable method to collect personal health data in healthy adults. For patients with chronic diseases, this would enable symptom monitoring and aid clinical decision making. Therefore, providing customized checklists to recommend smartwatches is beneficial. However, few studies have evaluated the practical functions of smartwatches and their influence on user acceptance. We aimed at developing a reliable tool to assess the quality of smartwatches from the users’ perspective.
Methods: To develop the smartwatch rating scale (SWRS), we conducted a comprehensive literature review as well as reviewed relevant websites. The SWRS includes 22 items for the usability (usability, functionality, safety, material, and display) and five items for the acceptance and adoption domain (satisfaction and intention). We measured the scale’s internal consistency and inter-rater reliability by evaluating seven smartwatches.
Results: The overall scale demonstrated an excellent level of internal consistency (Cronbach’s alpha = 0.91), with each subscale’s internal consistency above good level (0.74 ~ 0.92). Inter-rater reliability using intraclass correlation coefficients (ICC) was at good level (2-way random ICC = 0.82, 95% CI 0.09 – 0.97).
Conclusions: The SWRS is reliable, which can meet the need for assessment of smartwatch technology for utilizing in personal healthcare. Accounting for users’ perspectives will help make the most of technology without impairing the human aspects of care, this study can help consumers choose a smartwatch based on their preferences and provide guidelines for developing user-friendly wearable devices aimed at health behavior changes.
1. Baba NA, Baharudin AS, Alomari A (2019). Determinants of users’ intention to use smartwatch. J Theor Appl Inf Technol, 97(18):4738-50.
2. Guk K, Han G, Lim J, et al. (2019). Evolu-tion of wearable devices with real-time disease monitoring for personalized healthcare. Nanomaterials (Basel), 9(6):813.
3. Canalys, Worldwide wearable band market grew 65% in Q3 2019 while Asia Pacific doubled in volume. Available at https://canalys-prod-public.s3.eu-west 1.amazonaws.com/static/press_release/2019/CanalysWearablesPRQ32019v4.pdf . Accessed 07/15/2019.
4. Jung Y, Kim S, Choi B (2016). Consumer valuation of the wearables: The case of smartwatches. Comput. Human Behav, 63:899–905.
5. IDC. Shipments of Wearable Devices Reach 118.9 Million Units in the Fourth Quarter and 336.5 Million for 2019, According to IDC. https://www.telecomtv.com/content/idc/shipments-of-wearable-devices-reach-118-9-million-units-in-the-fourth-quarter-and-336-5-million-for-2019-according-to-idc-37984/
6. Chun JS. World smartwatch sales in Q2 12.3 million units 44% High Growth. Yonhap News Agency. 2019. https://www.yna.co.kr/view/AKR20190906024000017
7. Kheirkhahan M, Nair S, Davoudi A, et al (2019). A smartwatch-based framework for real-time and online assessment and mobility monitoring. J Biomed Inform, 89:29-40.
8. Lu TC, Fu CM, Ma HM, Fang CC, Tumer AM (2016). Healthcare applications of smart watches: A systematic review. Appl Clin Inform, 7(3):850-69.
9. Piwek L, Ellis DA, Andrews S, Joinson A (2016). The rise of consumer health wearables: Promises and barriers. PLoS Med, 13(2):1-9.
10. King CE, Sarrafzadeh M (2018). A survey of smartwatches in remote health monitor-ing. J Healthc Inform Res, 2(1-2):1-24.
11. Jia Y, Wang W, Wen D, Liang L, Gao L, Lei J (2018). Perceived user preferences and usability evaluation of mainstream weara-ble devices for health monitoring. Peer J, 25: 6:1-17.
12. van Helmond N, Freeman CG, Hahnen C, et al (2019). The accuracy of blood pres-sure measurement by a smartwatch and a portable health device. Hosp Pract, 47(4):211-5.
13. Wasserlauf J, You C, Patel R, Valys A, Albert D, Passman R (2019). Smartwatch per-formance for the detection and quantifi-cation of atrial fibrillation. Circ Arrhythm Electrophysiol, 12(6): e006834.
14. Dar R (2018). Effect of real-time monitoring and notification of smoking episodes on smoking reduction: A pilot study of a novel smoking cessation app. Nicotine Tob Res, 20(12): 1515–8.
15. Lu TC, Chang YT, Ho TW, et al (2019). Us-ing a smartwatch with real-time feedback improves the delivery of high-quality car-diopulmonary resuscitation by healthcare professionals. Resuscitation, 140: 16-22.
16. Rauschnabel PA, Krey N, Chuah S, Nguyen B, Lade S, Ramayah T (2016). Exploring the adoption of smartwatches. Digital En-terprise Computing, 39-48.
17. Strain T, Wijndaele K, Brage S (2019). Physi-cal activity surveillance through smartphone apps and wearable trackers: Examining the UK potential for national-ly representative sampling. JMIR MHealth UHealth, 7(1): 1-13.
18. Kim KJ (2016). Round or Square? How Screen Shape Affects Utilitarian and He-donic Motivations for Smartwatch Adop-tion. Cyberpsychol Behav Soc Netw,19(12): 733-9.
19. Anggraini N, Kaburuan ER, Wang G, Jayadi R (2019). Usability study and users’ per-ception of smartwatch: Study on Indone-sian customer. Procedia Comput Sci, 161: 1266-74.
20. Cheung ML, Chau KY, Lam MHS, Tse G, Ho KY, Flint SW (2019). Examining con-sumers' adoption of wearable healthcare technology: The role of health attributes. Int J Environ Res Public Health, 16(13): 1-16.
21. Gualtieri L, Rosenbluth S, Phillips J (2016). Can a free wearable activity tracker change behavior? The impact of trackers on adults in a physician-led wellness group. JMIR Res Protoc, 5(4): 1-8.
22. Mani Z, Chouk I (2017). Drivers of con-sumers? Resistance to smart products. J Market Manag, 33(1-2): 76–97.
23. Ha T, Beijnon B, Kim S, Lee S, Kim JH (2017). Examining user perceptions of smartwatch through dynamic topic mod-eling. Telematics and Informatics, 34(7): 1262–73.
24. Cecchinato ME, Cox AL, Bird J (2015). Smartwatches: The good, the bad and the ugly? Proceedings of the 33rd annual ACM conference extended abstracts on human factors in computing systems. A.C.M, 2133-8.
25. Nwosu AC, Quinn C, Samuels J, Mason S, Payne TR (2018). Wearable smartwatch technology to monitor symptoms in ad-vanced illness. BMJ Support Palliat Care, 8(2): 237.
26. Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M (2015). Mobile app rating scale: A new tool for assessing the quality of health mobile apps. JMIR MHealth UHealth, 3(1): 1-9.
27. Foster KR, Torous J (2019). The opportuni-ty and obstacles for smartwatches and wearable sensors. IEEE Pulse, 10(1): 22-5.
28. Chen CC, Shih HS (2014). A study of the acceptance of wearable technology for consumers: An analytical network pro-cess perspective. International Journal of the Analytic Hierarchy Process, 1-5.
29. Venkatesh V, Morris MG, Davis GB, Davis FD (2003). User acceptance of infor-mation technology: Toward a unified view. MIS Quarterly, 27(3): 425-478.
30. Dehghani M (2016). An assessment towards adoption and diffusion of smart wearable technologies by consumers: The cases of smart watch and fitness wristband prod-ucts. 27th ACM Conference on Hypertext and Social Media, CEUR Workshop Proceed-ings, 1628.
31. Canalys, Fitbit accounted for nearly half of global wearable band shipments in Q1 2014. Available at https://www.canalys.com/newsroom/fitbit-accounted-nearly-half-global-wearable-band-shipments-q1-2014 Ac-cessed 11/20/2018.
32. Canalys, Fitbit accounted for nearly half of global wearable band shipments in Q1. Available at https://www.canalys.com/newsroom/media-alert-fitbit-maintains-leadership-share-wearable-band-market-apple-watch-entrance Accessed 11/20/2018.
33. Mobihealthnews, Fitbit, Jawbone, Nike had 97 percent of fitness tracker retail sales. Available at http://www.mobihealthnews.com/28825/fitbit-jawbone-nike-had-97-percent-of-fitness-tracker-retail-sales-in-2013/ Accessed 11/23/2018.
34. Wen D, Zhang X, Liu X, Lei J (2017). Eval-uating the consistency of current main-stream wearable devices in health moni-toring: a comparison under free-living conditions. J Med Internet Res, 19(3): e68.
35. Gwet KL (2008). Computing inter-rater reli-ability and its variance in the presence of high agreement. Br J Math Stat Psychol, 61(1): 29-48.
36. Liljequist D, Elfving B, Skavberg Roaldsen K (2019). Intraclass correlation: A discus-sion and demonstration of basic features. PLoS One, 14(7): e0219854.
37. Zapf A, Castell S, Morawietz L, Karch A (2016). Measuring inter-rater reliability for nominal data - which coefficients and confidence intervals are appropriate?. BMC Med Res Methodol, 16: 1-10.
38. Sim J, Wright CC (2005). The Kappa Statis-tic in Reliability Studies: Use, Interpreta-tion, and Sample Size Requirements. Physl Ther, 85(3): 257–68.
39. McHugh ML (2012). Interrater reliability: the kappa statistic. Biochem Med, 22(3): 276-282.
2. Guk K, Han G, Lim J, et al. (2019). Evolu-tion of wearable devices with real-time disease monitoring for personalized healthcare. Nanomaterials (Basel), 9(6):813.
3. Canalys, Worldwide wearable band market grew 65% in Q3 2019 while Asia Pacific doubled in volume. Available at https://canalys-prod-public.s3.eu-west 1.amazonaws.com/static/press_release/2019/CanalysWearablesPRQ32019v4.pdf . Accessed 07/15/2019.
4. Jung Y, Kim S, Choi B (2016). Consumer valuation of the wearables: The case of smartwatches. Comput. Human Behav, 63:899–905.
5. IDC. Shipments of Wearable Devices Reach 118.9 Million Units in the Fourth Quarter and 336.5 Million for 2019, According to IDC. https://www.telecomtv.com/content/idc/shipments-of-wearable-devices-reach-118-9-million-units-in-the-fourth-quarter-and-336-5-million-for-2019-according-to-idc-37984/
6. Chun JS. World smartwatch sales in Q2 12.3 million units 44% High Growth. Yonhap News Agency. 2019. https://www.yna.co.kr/view/AKR20190906024000017
7. Kheirkhahan M, Nair S, Davoudi A, et al (2019). A smartwatch-based framework for real-time and online assessment and mobility monitoring. J Biomed Inform, 89:29-40.
8. Lu TC, Fu CM, Ma HM, Fang CC, Tumer AM (2016). Healthcare applications of smart watches: A systematic review. Appl Clin Inform, 7(3):850-69.
9. Piwek L, Ellis DA, Andrews S, Joinson A (2016). The rise of consumer health wearables: Promises and barriers. PLoS Med, 13(2):1-9.
10. King CE, Sarrafzadeh M (2018). A survey of smartwatches in remote health monitor-ing. J Healthc Inform Res, 2(1-2):1-24.
11. Jia Y, Wang W, Wen D, Liang L, Gao L, Lei J (2018). Perceived user preferences and usability evaluation of mainstream weara-ble devices for health monitoring. Peer J, 25: 6:1-17.
12. van Helmond N, Freeman CG, Hahnen C, et al (2019). The accuracy of blood pres-sure measurement by a smartwatch and a portable health device. Hosp Pract, 47(4):211-5.
13. Wasserlauf J, You C, Patel R, Valys A, Albert D, Passman R (2019). Smartwatch per-formance for the detection and quantifi-cation of atrial fibrillation. Circ Arrhythm Electrophysiol, 12(6): e006834.
14. Dar R (2018). Effect of real-time monitoring and notification of smoking episodes on smoking reduction: A pilot study of a novel smoking cessation app. Nicotine Tob Res, 20(12): 1515–8.
15. Lu TC, Chang YT, Ho TW, et al (2019). Us-ing a smartwatch with real-time feedback improves the delivery of high-quality car-diopulmonary resuscitation by healthcare professionals. Resuscitation, 140: 16-22.
16. Rauschnabel PA, Krey N, Chuah S, Nguyen B, Lade S, Ramayah T (2016). Exploring the adoption of smartwatches. Digital En-terprise Computing, 39-48.
17. Strain T, Wijndaele K, Brage S (2019). Physi-cal activity surveillance through smartphone apps and wearable trackers: Examining the UK potential for national-ly representative sampling. JMIR MHealth UHealth, 7(1): 1-13.
18. Kim KJ (2016). Round or Square? How Screen Shape Affects Utilitarian and He-donic Motivations for Smartwatch Adop-tion. Cyberpsychol Behav Soc Netw,19(12): 733-9.
19. Anggraini N, Kaburuan ER, Wang G, Jayadi R (2019). Usability study and users’ per-ception of smartwatch: Study on Indone-sian customer. Procedia Comput Sci, 161: 1266-74.
20. Cheung ML, Chau KY, Lam MHS, Tse G, Ho KY, Flint SW (2019). Examining con-sumers' adoption of wearable healthcare technology: The role of health attributes. Int J Environ Res Public Health, 16(13): 1-16.
21. Gualtieri L, Rosenbluth S, Phillips J (2016). Can a free wearable activity tracker change behavior? The impact of trackers on adults in a physician-led wellness group. JMIR Res Protoc, 5(4): 1-8.
22. Mani Z, Chouk I (2017). Drivers of con-sumers? Resistance to smart products. J Market Manag, 33(1-2): 76–97.
23. Ha T, Beijnon B, Kim S, Lee S, Kim JH (2017). Examining user perceptions of smartwatch through dynamic topic mod-eling. Telematics and Informatics, 34(7): 1262–73.
24. Cecchinato ME, Cox AL, Bird J (2015). Smartwatches: The good, the bad and the ugly? Proceedings of the 33rd annual ACM conference extended abstracts on human factors in computing systems. A.C.M, 2133-8.
25. Nwosu AC, Quinn C, Samuels J, Mason S, Payne TR (2018). Wearable smartwatch technology to monitor symptoms in ad-vanced illness. BMJ Support Palliat Care, 8(2): 237.
26. Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M (2015). Mobile app rating scale: A new tool for assessing the quality of health mobile apps. JMIR MHealth UHealth, 3(1): 1-9.
27. Foster KR, Torous J (2019). The opportuni-ty and obstacles for smartwatches and wearable sensors. IEEE Pulse, 10(1): 22-5.
28. Chen CC, Shih HS (2014). A study of the acceptance of wearable technology for consumers: An analytical network pro-cess perspective. International Journal of the Analytic Hierarchy Process, 1-5.
29. Venkatesh V, Morris MG, Davis GB, Davis FD (2003). User acceptance of infor-mation technology: Toward a unified view. MIS Quarterly, 27(3): 425-478.
30. Dehghani M (2016). An assessment towards adoption and diffusion of smart wearable technologies by consumers: The cases of smart watch and fitness wristband prod-ucts. 27th ACM Conference on Hypertext and Social Media, CEUR Workshop Proceed-ings, 1628.
31. Canalys, Fitbit accounted for nearly half of global wearable band shipments in Q1 2014. Available at https://www.canalys.com/newsroom/fitbit-accounted-nearly-half-global-wearable-band-shipments-q1-2014 Ac-cessed 11/20/2018.
32. Canalys, Fitbit accounted for nearly half of global wearable band shipments in Q1. Available at https://www.canalys.com/newsroom/media-alert-fitbit-maintains-leadership-share-wearable-band-market-apple-watch-entrance Accessed 11/20/2018.
33. Mobihealthnews, Fitbit, Jawbone, Nike had 97 percent of fitness tracker retail sales. Available at http://www.mobihealthnews.com/28825/fitbit-jawbone-nike-had-97-percent-of-fitness-tracker-retail-sales-in-2013/ Accessed 11/23/2018.
34. Wen D, Zhang X, Liu X, Lei J (2017). Eval-uating the consistency of current main-stream wearable devices in health moni-toring: a comparison under free-living conditions. J Med Internet Res, 19(3): e68.
35. Gwet KL (2008). Computing inter-rater reli-ability and its variance in the presence of high agreement. Br J Math Stat Psychol, 61(1): 29-48.
36. Liljequist D, Elfving B, Skavberg Roaldsen K (2019). Intraclass correlation: A discus-sion and demonstration of basic features. PLoS One, 14(7): e0219854.
37. Zapf A, Castell S, Morawietz L, Karch A (2016). Measuring inter-rater reliability for nominal data - which coefficients and confidence intervals are appropriate?. BMC Med Res Methodol, 16: 1-10.
38. Sim J, Wright CC (2005). The Kappa Statis-tic in Reliability Studies: Use, Interpreta-tion, and Sample Size Requirements. Physl Ther, 85(3): 257–68.
39. McHugh ML (2012). Interrater reliability: the kappa statistic. Biochem Med, 22(3): 276-282.
| Files | ||
| Issue | Vol 52 No 1 (2023) | |
| Section | Original Article(s) | |
| DOI | https://doi.org/10.18502/ijph.v52i1.11668 | |
| Keywords | ||
| Smartwatches Mobile health Wearable technology Healthcare Questionnaires | ||
| Rights and permissions | |
|
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. |
How to Cite
1.
Lee S-K, Kim GY, Seo EJ, Son Y-J. Initial Development of User-Based Quality Evaluation Questionnaire of Smartwatch Technology for Applying to Healthcare. Iran J Public Health. 2023;52(1):78-86.
