2021 36th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW)
2021 36th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) | 978-1-6654-3583-3/21/$31.00 ©2021 IEEE | DOI: 10.1109/ASEW52652.2021.00039
ACSIMA: A Cyber Security Index for Mobile
Health Apps
Hamza Sellak1 , Mohan Baruwal Chhetri1 , Marthie Grobler1 , Kristen Moore1 , 2
1 CSIRO’s
Data61, Australia
Security Cooperative Research Centre, Australia
Email: {hamza.sellak, mohan.baruwalchhetri, marthie.grobler, kristen.moore}@data61.csiro.au
2 Cyber
Abstract—Mobile health (mHealth) apps can make healthcare
more accessible and affordable by enabling users to perform
a range of self-service activities to manage their own health
and wellbeing. However, the mHealth app market is largely
unregulated, potentially exposing app users to substantial cyber
security risks. In this exploratory study, we present the Australian
Cyber Security Index for mHealth Apps (ACSIMA) as a curated
cyber security checklist that can guide the assessment of existing
mHealth apps as well as the design/development of new apps.
In contrast to existing mHealth app assessment frameworks,
ACSIMA (1) focuses exclusively on cyber security, (2) takes a
multi-stakeholder approach to cyber security assessment, and
(3) is specific to the Australian digital health context. ACSIMA is
aimed at raising the level of cyber security acuity among different
stakeholder groups, including app users and app developers, and
can be considered a first step towards the provision of reliable
and trustworthy digital health services. We validate ACSIMA’s
usability and practicability through an online survey with the
different stakeholder groups. The study finds that different
stakeholder groups exhibit different levels of familiarity and
importance towards the ACSIMA checklist, validating the need
for raising the awareness of cyber security concerns in mHealth
apps across all stakeholder groups.
Index Terms—mHealth apps, security, privacy, checklist
I. I NTRODUCTION
Advances in smartphone technology and the digitisation
of healthcare is enabling more people to use mobile health
(mHealth) apps to actively self-manage their health and wellbeing. There are two broad categories of mHealth apps: (a)
medical apps used for the purpose of prevention, diagnosis,
and treatment of diseases, and (b) non-medical apps related to
lifestyle, fitness and wellbeing. In Australia, the Therapeutic
Goods Administration (TGA) is the regulatory authority for
therapeutic goods, including medical devices, and covers Software as a Medical Device (SaMD), i.e., software that by itself
functions as a medical device [1]. Such SaMDs are assessed
for compliance against a list of essential principles [2], and
audited for conformity to the International Organization for
Standardization (ISO) medical devices regulation requirements
(ISO-13485).
There are specific guidelines to help consumers understand
and manage their risks and responsibilities when using the
SaMDs [3]. However, the majority of mHealth apps are
lifestyle and fitness apps that are considered “simply sources
of information, or tools to manage a healthy lifestyle” and
therefore do not fall under the TGA’s remit [1]. The market for
978-1-6654-3583-3/21/$31.00 ©2021 IEEE
DOI 10.1109/ASEW52652.2021.00039
such non-SaMD mHealth apps is therefore largely unregulated.
While published apps are required to meet the quality criteria
of the app stores through which they are offered, compliance
to TGA regulations is not mandatory. Consequently, many
mHealth apps are not subjected to a thorough cyber security
assessment and validation process, and can potentially expose
app users to substantial security and privacy risks, as highlighted by a recent study of 18,000 popular free apps available
on Google Play Store. This study found that ∼50% of the
apps were missing a privacy policy even though ∼70% of
these were processing personally identifiable information [4].
Researchers have attributed this to apptimism [5], i.e., an
implicit trust in smartphones and everything that runs on them.
Our preliminary research shows that even though multiple
government agencies in Australia are focusing their attention
on the privacy and security of health data [6]–[8], there
is currently no regulation in place to ensure that mHealth
apps adhere to security and privacy principles to keep their
users safe from cyber insecurities. As outlined in Section
II-B, there are several frameworks that support the design
and assessment of mHealth apps; however, they either do
not sufficiently address security and privacy, or only target
particular stakeholder groups.
To address this gap in the intersection between digital
health and cyber security practices, our exploratory research
study puts forward the Australian Cyber Security Index for
mHealth Apps (ACSIMA) as a comprehensive checklist that
can help Australian stakeholders, specifically app users, app
developers, healthcare providers, and policy makers, to use,
design, and develop mHealth apps with more confidence. The
index was compiled after an extensive review of published
literature on digital health, security and privacy, as well as
existing Australian and international standards, regulations
and guidelines to create a criteria list that is customised for
different stakeholder groups. Following this, an online survey
was conducted to answer the following research questions:
RQ1: How familiar are the different stakeholder groups with
the cyber security criteria included in ACSIMA?, and RQ2:
What is their perceived importance of the cyber security
criteria included in ACSIMA? The main finding from the
survey is that different stakeholder groups show different levels
of familiarity and perceived importance towards the ACSIMA
checklist. These perceptions are likely shaped by their personal
and professional experiences. However, the familiarity is low
145
Authorized licensed use limited to: Nazarbayev University. Downloaded on April 10,2023 at 05:25:27 UTC from IEEE Xplore. Restrictions apply.
across all stake holder groups, validating the overall need for
raising the awareness of cyber security concerns in mHealth
apps across all stakeholder groups.
The paper is structured as follows. Section II presents an
overview of related work in the context of mHealth apps’
standards, regulations and assessment frameworks. Section
III describes the methodology followed in compiling and
refining ACSIMA’s criteria list. Section IV details the results
from our multi-stakeholder study, whilst Section V provides
a discussion on our findings. Section VI concludes the paper
with an outlook on future work.
II. R ELATED W ORK
In this section, we briefly overview the key standards
and regulations applicable to mHealth apps, and prominent
mHealth assessment frameworks proposed in the literature.
A. Standards and Regulations
mHealth apps that cater to health and lifestyle are largely
unregulated in major digital health markets such as the United
States (US), Europe and Australia. In the US, the Food and
Drug Administration does not regulate health and wellness
apps that are not intended for medical use despite an increasing
push for it to play a much stronger role [9]. In Australia, the
TGA does not regulate health and lifestyle apps and software
that do not fit their definition of a medical device [1].
There are some progress in Europe with the European
Commission facilitating the Privacy Code of Conduct [10]
to provide app developers with practical guidance on data
protection principles when developing mHealth apps. However, it is yet to achieve formal approval of European data
authorities. Similarly, the Office of the Australian Information Commissioner (OAIC) promotes the Australian Privacy
Principles (APPs) [11] to govern the standards, rights, and
obligations around the handling of personal information and
are therefore very important in the context of mHealth apps.
It also provides a guide for app developers operating in the
Australian market to comply with Australian privacy law and
best practices [12]. Also, the Australian Commission on Safety
and Quality in Healthcare has recently released the National
Safety and Quality Digital Mental Health Standards [13] in
partnership with key stakeholders, including mental health
service users, consumers, carers, families, and technical experts. The standards apply to digital mental health services
and cover technical governance including privacy, security, and
transparency of digital systems, but is yet to extend to cover all
mHealth apps. Similarly, the Australian Digital Health Agency,
which plays a key role in the adoption and use of digital
health services and technologies, provides only basic guidance
to users on how to stay private and secure online but none
regarding the use of mHealth apps.
Notwithstanding the existence of different government entities and their ongoing efforts looking at different aspects
of digital health, the mHealth app market is still largely
unregulated. Therefore, there is a strong need for tools and
frameworks to help users navigate the mHealth app market.
B. mHealth Assessment Frameworks
Several cyber security assessment frameworks and tools
have been proposed to cater for the needs of different
stakeholders. The Interactive Mobile App Review Toolkit
(IMART) [14] is a framework for reviewing mHealth apps
and is aimed at assisting clinicians select healthcare apps.
However, it offers little consideration for security and privacy.
Similarly, the American Psychiatric Association (APA) App
Evaluation Framework [15], which is targeted at patients with
psychiatric conditions and clinicians treating them, provides
cursory treatment of cyber security concerns. The Technology
Evaluation and Assessment Criteria for Health Apps (TEACHApps) [16] is a structured 4-step process for evaluating
mHealth apps built on top of the APA framework and is
targeted at hospitals and health care facilities. The GeoHealth
Privacy Impact Assessment (PIA) framework [17], which identifies and evaluates the potential privacy risks of mHealth data
collection systems, is targeted at community health workers.
The Mobile App Rating Scale (MARS) [18], [19] is a tool
for classifying and assessing the quality of mHealth apps; it
is targeted at researchers, developers and health professionals. The Mobile App Development and Assessment Guide
(MAG) [20] provides a set of criteria that can help health
care providers, developers, patients, and other stakeholders
to guide the development and assessment of mHealth apps.
THESIS [21] is a rating tool to objectively measure the risks
and benefits of mHealth apps. This tool can be used by policy
makers, app developers, patients and healthcare providers.
Finally, the Organisation for the Review of Care and Health
Applications - 24 Questions (ORCHA-24) framework [22]
consists of 24 app assessment criteria that are derived from
published standards on mHealth apps, data privacy and user
experience and presented as ‘yes’ or ‘no’ questions without
the requirement for scaling or value judgements.
There is a lot of diversity in existing assessment frameworks; some are targeted at specific stakeholders (e.g., [14]–
[16]) while others cater to multiple stakeholders (e.g., [17],
[18], [20]–[22]). However, most of them fail to sufficiently
address cyber security concerns (e.g., [18]), while some provide basic coverage only (e.g., [15], [21], [22]).
C. Distinction from Prior Work
All stakeholders have a shared responsibility in addressing
the cyber security issues associated with medical devices,
and by extension, mHealth apps. Yet, a key limitation of
existing mHealth app assessment frameworks is that they fail
to acknowledge that different stakeholders may have different
cyber security concerns, and may be differently impacted
by them. ACSIMA addresses this shortcoming by providing
customised cyber security checklists for the identified key
stakeholder groups. Furthermore, ACSIMA complements and
covers existing Australian standards and regulations.
III. M ETHODOLOGY
The main objective of this study was to generate a curated
cyber security checklist that can be used as an index to guide
146
Authorized licensed use limited to: Nazarbayev University. Downloaded on April 10,2023 at 05:25:27 UTC from IEEE Xplore. Restrictions apply.
TABLE I: Source Documents for Index Generation
Source Document
Literature
Baumel et al. [25]
Benjumea at al. [26]
Bandaronek et al. [27]
Camacho et al. [16]
Huckvale et al. [28]
Huckvale et al. [29]
Hutton et al. [30]
Karekla et al. [31]
Llorens-Vernet and Miró [20]
O’Loughlin et al. [32]
Scott et al. [33]
Other
ADHA [34]
ADHA [35]
ANDHealth [24]
APP [11]
IMDRF [23]
NSQDMH [13]
OAIC [12]
TGA [1]
in the process. The initial criteria list was refined through
multiple iterations resulting in a final list of 50 criteria.
Type
Systematic review
Scoping review
App review
Primary research
App review
App review
App review
Primary research
Systematic review
App Review
Literature review
B. Index Customisation
Following the finalisation of the criteria list, a customised
version was generated for each stakeholder group by considering which criteria would be more relevant and applicable
to each group. Thereafter, the Delphi method was used to
scale down the customised lists to a more manageable number
with the research team voting for the inclusion/exclusion
of each criterion for each stakeholder group. The following
specifications were considered for each group:
Guide
Guide
Report
Law
Guide
Standard
Guide
Regulation
•
•
the evaluation of existing mHealth apps and the design and
development of new ones. This section provides an overview
of how the checklist was generated and validated for usability
and applicability.
•
A. Index Generation
The cyber security checklist was generated in two phases
using two main sources of information. In the first phase, an
initial criteria list was drafted by searching through published
academic literature, using Web of Science and Google Scholar,
for studies that focused on identifying standards for mHealth
apps. Starting with an initial set of articles, snowballing was
used to identify further relevant articles. Key terms used in
the search include mobile health app, mHealth app, privacy,
security, cyber security, evaluation and assessment framework.
In the second phase, different Australian and International
cyber security regulations and other online guidelines were
examined to complement the criteria list. This includes the
regulations provided by TGA [1], the guidelines provided
by ADHA, the NSQDMH standards from the Australian
Commission on Safety and Quality in Healthcare [13], the
guidelines from the International Medical Device Regulators
Forum (IMDRF) [23], and research reports by Australia’s
National Digital Health Initiative (ANDHealth) [24].
The initial list compiled after analysing the sources listed
in Table I had 144 criteria. It covers a broad spectrum of
cyber security practices, guidelines and recommendations,
and is applicable to a range of stakeholder groups. These
criteria were separated into two classes (CLs): CL1: Design
and Development criteria, which may be considered during
the design and development phases of mHealth apps, and
CL2: Evaluation criteria, which may be considered during the
evaluation of existing mHealth apps. A thorough review of
this list was conducted by the research team to remove any
duplicate criteria and to merge similar criteria. 26 redundant
criteria were removed, and a further 86 criteria were merged
•
•
•
Policy makers (PM) are responsible for formulating
policies in health and/or cyber security. The customised
list should focus on criteria that belong to both CL1 and
CL2, balancing both technical1 and non-technical criteria.
Health Professionals (HP) provide health care treatment
and advice based on formal training and experience. The
customised list should focus on criteria that belong to
CL2 and balance both technical/non-technical criteria.
Researchers (R) conduct research in the digital health
and/or cyber security domains. The customised list should
focus on criteria that belong to both CL1 and CL2, and
put more emphasis on technical criteria.
Software Engineers/App Developers (SE/AD) design
and develop software and digital (health) solutions. The
customised list should focus on criteria that belong to
CL1, and put more emphasis on technical criteria.
Consumers (C) refer to those who are currently using or
have previously used an mHealth app. The customised list
should focus on criteria that belong to CL2, and should
avoid technical criteria.
Others (O) that do not fall under any of the previous
stakeholder groups. The customised list should focus on
criteria that belong to CL2, and avoid technical criteria.
C. Index Validation
An exploratory user study was conducted to validate the
perceived usability and applicability of the developed cyber
security index. An online survey was conducted2 targeting
the identified stakeholder groups and using the corresponding
customised criteria list. Table II provides a summary of the
different sections of the survey and the corresponding questions for each stakeholder group. Participants were specifically
asked to comment on their familiarity and perceived importance of each criterion. The multi-stakeholder feedback was
incorporated to ensure the relevance and usability of the index
within the targeted stakeholder groups.
1 Technical criteria describe processes related to cyber security and/or software engineering such as encryption, security controls, and risk management.
2 Following approval by the organisational ethics review board, the online
survey was conducted through SurveyMonkey3 during December 2020 and
January 2021.
147
Authorized licensed use limited to: Nazarbayev University. Downloaded on April 10,2023 at 05:25:27 UTC from IEEE Xplore. Restrictions apply.
TABLE II: Survey Section and Question Overview
Stakeholder
Group
PM
HP
R
SE/AD
C
O
TABLE III: Sociodemographic Characteristics of the Stakeholder Groups
Section [Question(s)]
Sociodemographics [Q1-Q3]
Expertise [Q4]
Cyber security risks awareness [Q9]
Cyber security & mHealth apps [Q10]
Customised ACSIMA [Q11]
Sociodemographics [Q1-Q3]
Expertise [Q5]
Cyber security risks awareness [Q9]
Cyber security & mHealth apps [Q10]
Customised ACSIMA [Q12]
Sociodemographics [Q1-Q3]
Expertise [Q6]
Cyber security risks awareness [Q9]
Cyber security & mHealth apps [Q10]
Customised ACSIMA [Q13]
Sociodemographics [Q1-Q3]
Expertise [Q7]
Cyber security risks awareness [Q9]
Cyber security & mHealth apps [Q10]
Customised ACSIMA [Q14]
Sociodemographics [Q1-Q3]
Expertise [Q8]
Cyber security risks awareness [Q9]
Cyber security & mHealth apps [Q10]
Customised ACSIMA [Q15]
Sociodemographics [Q1-Q3]
Cyber security risks awareness [Q9]
Cyber security & mHealth apps [Q10]
Customised ACSIMA [Q16]
PM
5 (4%)
HP
16 (12%)
PM
5 (4%)
HP
16 (12%)
18-25
26 (20%)
26-35
40 (30%)
Stakeholder Group
R
SE/AD
C
O
42 (32%)
34 (25%)
17 (13%)
18 (14%)
Gender
R
SE/AD
C
O
42 (32%)
34 (25%)
17 (13%)
18 (14%)
Age Group
36-45
46-55
56-65
>65
PNTA
39 (29%) 8 (6%) 10 (8%) 6 (5%) 3 (2%)
TABLE IV: Expertise of the Stakeholder Groups
PM
Have you ever been part of a team that prepared a policy related to
(digital) health and/or cyber security?
No
2 (50%)
Yes, I am/was a team leader
2 (50%)
Yes, I am/was a team member
1 (25%)
HP
What is your exact role as a health professional?
Medical specialist
1 (6%)
Allied health professional
11 (69%)
Other
5 (31%)
R
What are your research focus areas and expertise?
Health research
12 (30%)
Digital health
14 (35%)
Cyber security
21 (53%)
Computer science (in general)
14 (35%)
Other
2 (5%)
SE/AD
How do you describe your expertise as a software engineer/app
developer?
I am/was involved in the development of at least one
7 (22%)
mHealth app (or similar digital health services)
I am/was involved in the development of non-health23 (72%)
related mobile apps
Other
6 (19%)
C
Have you ever used any technology medium to help you achieve
your health and/or wellbeing goals?
No
1 (6%)
Yes, I used an activity tracker watch
14 (82%)
Yes, I used a medical wearable device
1 (6%)
Yes, I joined an online health program or group about
2 (12%)
a health issue
Yes, I entered information on a website about my
13 (76%)
physical activity, nutrition, sleep and/or wellbeing
Yes, I searched for information about a disease, health
13 (76%)
problem or a doctor
Yes, I used a smart home environment for healthcare
2 (12%)
and assistance
D. Limitations
The purpose of this research study was to develop an index
for mHealth apps that focuses exclusively on cyber security,
considers multiple stakeholders and is mainly applicable to the
Australian digital health context. An important consideration
in this research is that individuals’ perceptions of security and
privacy are influenced by their own personal and professional
experiences, their technological innovations and their role
within the digital domain. All of these aspects can therefore
play a role in influencing the participants’ responses to our
survey. We found that there was not enough representation
across all the different stakeholder groups. In particular, the
policy makers group was under represented. We are therefore
unable to define conclusive findings on the perceived usability
and applicability of the index for this specific group.
IV. R ESULTS
This section presents the results from our exploratory survey. We report on the results of each survey section using
statistical metrics such as count (N ), percentage (%), mean
(µ), standard deviation (σ), and coefficient of variation. We
also summarize the key findings.
A. Sociodemographics
All participants were required to complete a set of sociodemographic questions (Q1-Q3 in Table II) pertaining to
gender, age and self-classification. Depending upon the participants’ response to self-classification, logic branching was
applied to direct them to subsequent questions related to their
selected stakeholder group. Table III provides the summary
statistics of the 132 participants’ responses.
B. Expertise
All participants except those in the other group were
required to respond to the expertise section of the survey
(Q4-Q8 in Table II). The key findings include: (1) most
of the stakeholder groups demonstrated a connection with
digital health and/or mHealth apps; (2) despite their small
number, most policy makers indicated that they have been
involved in the making of at least one policy related to
(digital) health and/or cyber security; (3) one role (allied health
148
Authorized licensed use limited to: Nazarbayev University. Downloaded on April 10,2023 at 05:25:27 UTC from IEEE Xplore. Restrictions apply.
100
60
40
20
0
No, not at all
Don t know/can t say
Yes, definitely
Yes, to some extent
80
percentage
80
percentage
100
Poor
Fair
Good
Very good
60
40
20
PM
HP
R
SE/AD
Participant group
C
0
O
Fig. 1: Distribution of Cyber Security Risk Awareness.
PM
HP
R SE/AD C
Participant group
O
Fig. 2: Distribution of the Assumption that Cyber Security is
Built into mHealth apps.
professional) was dominant among health professionals with
zero participation from general practitioners, dentists or oral
health professionals, complementary medicine practitioners,
and other medical specialities; (4) there was relatively equal
participation from researchers in cyber security, digital health
and health research with many specialising in two or more
research domains; and (4) participants from the consumer
stakeholder group indicated that all of them are tech-savvy and
had medium interactions with different technology mediums
that help to achieve health and/or wellbeing goals. Table IV
provides the summary statistics of the responses of the 132
participants.
C. Cyber Security Risk Awareness
All participants were asked a question related to cyber
security risk awareness (Q9 in Table II). Around 95% of the
participants (N = 125) answered this question with 40%
(N = 50) rating their current awareness of cyber security
risks as “Good”, 34% (N=43) as “Fair”, 16% as “Very
good”, and 10% (N=12) as “Poor”. None of the participants
rated their current awareness of cyber security risks as “Very
poor”. Figure 1 presents the overall distribution of ratings
for each stakeholder group. At a group level, ratings from
the software engineers/app developers and researchers were
slightly higher than the rest of the groups. The overall rating
provided by health professionals can be considered closer to
“Poor”. The majority of consumers indicated that they have
overall knowledgeable awareness of cyber security risks.
D. Cyber Security and mHealth Apps
All participants were asked if they thought that cyber
security is built into all mHealth apps (Q10 in Table II). 95%
of participants answered the question with 41% (N = 51)
indicating that they think cyber security is built into all
available mHealth apps to some extent, 38% (N = 48)
disagreed with this assumption, 16% (N = 16) indicated
that they do not know or cannot say, and only 5% (N = 6)
strongly supported the assumption (i.e., answered with “Yes,
definitely”). Figure 2 presents the distribution of responses of
each stakeholder group to the assumption that cyber security
is built into all available mHealth apps.
Key conclusions from the responses to Q10 are: (1) there
was consensus that cyber security is not naturally built into
all mHealth apps; (2) health professionals were more supportive of the assumption, whereas researchers expressed less
acceptance of the assumption in comparison to the rest of
the stakeholder groups; (3) almost half of the software engineers/app developers indicated that they “don’t know/can’t
say” which is justified by the fact that only 22% (N = 7)
are/were involved in the development of at least one mHealth
app or similar digital health services (see Section IV-B); and
(4) consumers and other stakeholders had relatively similar
and equal distributions over both “Yes” and “No” answers.
E. Customised ACSIMA
The customised ACSIMA section was presented to all stakeholder groups, respectively as Q11-16 (see Table II). Only
47% of the participants completed this section of the survey.
Since we did not have enough representation from the policy
makers group, we excluded them in this section.
We compiled a separate criteria list for each stakeholder
group and asked the participants to consider: (a) how familiar
they are with each criterion using a 0–5 scale (0: Not familiar
at all – 5: Extremely familiar) and (b) how important each
criterion is to them using a 0–5 scale (0: Not important at all –
5: Extremely important). Table A1 presents the comprehensive
criteria list. Table V lists the top 10 criteria according to each
stakeholder group, ranked in terms of familiarity and perceived
importance. This table gives an indication of how relevant
stakeholders consider the criteria to be for their use of the
apps.
Table VI shows the overall mean, standard deviation, and
coefficient of variation given by each stakeholder group to their
criteria list. If two criteria have the same mean, the one with
the lowest standard deviation4 is considered better. The last
4 The standard deviation describes how much an individual measurement
should be expected to deviate on average from the mean. A smaller standard
deviation means greater consistency, predictability and quality.
149
Authorized licensed use limited to: Nazarbayev University. Downloaded on April 10,2023 at 05:25:27 UTC from IEEE Xplore. Restrictions apply.
TABLE V: Top 10 Criteria based on Familiarity and Perceived
Importance (for each Stakeholder Group)
HPF
C8
C23
C14
C6
C15
C25
C16
C1
C4
C24
HPI
C4
C17
C6
C23
C14
C15
C25
C19
C27
C26
RF
C23
15
C9
C8
C34
C32
C16
C33
C12
C6
RI
C15
C23
C29
C4
C6
C34
C32
C9
C13
C39
SE/ADF
C33
C8
C12
C17
C41
C32
C42
C43
C15
C16
SE/ADI
C9
C32
C33
C12
C13
C16
C4
C15
C6
C17
CF
C16
C24
C25
C13
C47
C9
C12
C8
C11
C23
CI
C13
C16
C9
C23
C24
C17
C25
C15
C21
C8
OF
C23
C25
C47
C12
C8
C9
C16
C6
C10
C24
OI
C23
C8
C9
C25
C4
C21
C24
C6
C17
C13
TABLE VI: Mean, Standard Deviation, and Coefficient of
Variation of Responses (for each Stakeholder Group)
Familiarity
µF
HP
2.935
R
3.232
SE/AD 3.106
C
2.995
O
3.108
σF
1.164
1.452
1.675
1.511
1.278
Importance
CVF
0.396
0.449
0.539
0.504
0.411
µI
4.447
4.087
3.841
3.695
3.869
σI
0.806
1.035
1.221
1.430
1.065
|µF − µI | |σF − σI |
CVI
0.181
0.253
0.317
0.387
0.275
1.511
0.854
0.735
0.700
0.769
0.807
0.416
0.454
0.080
0.212
two columns (|µF − µI | and |σF − σI |) show the discrepancy
between overall expressed familiarity and perceived importance. The best and worst results of each column are highlighted in bold font and underlined, respectively. Researchers
gave the highest overall rating regarding familiarity with criteria (µF = 3.232), health professionals gave the lowest overall
rating (µF = 2.935) and showed less variation and greater
consistency (σF = 1.164), whereas software engineers/app
developers have the highest variation and less consistency
(σF = 1.675) in comparison to the rest of the stakeholder
groups. In terms of perceived importance, health professionals provided the highest overall rating (µI = 4.447) with
less variation and greater consistency (σI = 0.806), whereas
consumers provided the lowest overall rating (µI = 3.695)
with higher variation and less consistency (σI = 1.430). These
results are confirmed by each stakeholder group CV5 values
for both familiarity (CVF ) and perceived importance (CVI ).
Consumers followed by software engineers/app developers
(|µF − µI | = 0.70 and |µF − µI | = 0.735, respectively)
demonstrated less discrepancy, whereas health professionals
demonstrated the highest discrepancy (|µF − µI | = 1.511).
These results are confirmed by each stakeholder group’s
|σF − σI | value from the last column (i.e., the smaller the
value the greater the consistency between the familiarity and
perceived importance).
The key findings to consider from this section are: (1)
across all stakeholder groups, participants rated the criteria
that were of a non-technical or legal/regulatory nature higher
than the more technical criteria; (2) criteria that focus on
privacy, such as C4, C6, C9 and C23 (see Table A1) were
perceived as being more important by all stakeholder groups
– they appeared in the top five of at least three of the
5 The coefficient of variation shows the extent of variability in relation to
the population mean. A smaller coefficient of variation means less variability.
customised lists; (3) the inconsistency between familiarity and
perceived importance in some stakeholder groups is likely due
to their lack of cyber security awareness – e.g., the majority of
researchers stated that they specialised in cyber security and
expressed higher familiarity with the criteria, whereas health
professionals expressed less familiarity and higher perceived
importance which aligns with their overall “Poor” to “Fair”
cyber security risks awareness and their strong support of
the assumption that cyber security is built into all available
mHealth apps; and finally, (4) the less familiarity participants
have with a criterion, the higher their perceived importance of
that criterion – e.g., health professionals.
V. D ISCUSSION
In this study, we developed ACSIMA, a cyber security index
for mHealth apps, by compiling a comprehensive set of criteria
and then iteratively refining it to generate a curated list. We
created customised versions of the list for each of the target
stakeholder groups and performed an online survey to explore
the following two questions:
• RQ1: How familiar are the different stakeholder groups
with the key cyber security criteria included in ACSIMA?
Some stakeholder groups are more familiar and aware
of the security and privacy criteria included in ACSIMA.
For example, the cyber security researchers and software
engineers/app developers had a higher mean familiarity
rating compared to the consumers and health professionals. However, the mean familiarity ratings for all
stakeholder groups were low indicating that there is an
overall need for raising the awareness of cyber security
concerns in mHealth apps across all stakeholder groups.
• RQ2: What is their perceived importance of the key
cyber security criteria included in ACSIMA? Even though
the different stakeholder groups indicated different levels
of familiarity with the cyber security concerns in mHealth
apps, they all considered the criteria included in their customised checklists to be important. As expressed by one
of the participants, “there were many criteria that I was
not formally aware of, but which I thought were important
once brought to my attention”. This again highlights the
need for increased awareness of cyber security concerns
across all stakeholder groups.
ACSIMA aims to help designers and developers to deliver
mHealth apps that are reliable and supported by secure and
privacy-aware technology; and educate end-users about the potential cyber security risks of mHealth apps. The development
of ACSIMA is a precursor to transforming the state-of-theart of digital health and wellbeing interventions. The wider
implementation of the index has the potential to increase cyber
security awareness among all stakeholders, leading to increase
stakeholder engagement, which in turn would enable enhanced
security, privacy and transparency in mHealth apps.
VI. C ONCLUSION
Mobile health (mHealth) apps have the potential to make
healthcare more accessible and affordable for everyone, but
150
Authorized licensed use limited to: Nazarbayev University. Downloaded on April 10,2023 at 05:25:27 UTC from IEEE Xplore. Restrictions apply.
the current market is largely unregulated. Most apps available
in major app stores are not subjected to a thorough cyber security assessment and validation process, leaving the end-users
vulnerable to cyber insecurities. We developed the Australian
Cyber Security Index for Mobile Health Apps (ACSIMA) as a
first step towards the transition to trustworthy mHealth apps. In
contrast to existing assessment frameworks, ACSIMA (1) exclusively focuses on cyber security, (2) provides customised
checklists for different stakeholders groups in the digital health
domain, and (3) is specific to the Australian context. Although
ACSIMA is currently in a prototype stage, still to be considered
by representative stakeholder groups in a large collaborative
setting, it has the potential to not only educate about digital
health services, their benefits and potential cyber security risks,
but also to build trust and increase engagement and adherence
to personal health and wellbeing management services.
ACKNOWLEDGMENT
This research was financially supported by CSIRO’s Precision Health Future Science Platform.
R EFERENCES
[1] Therapeutic Goods Administration, “Regulation of software based medical devices,” https://tinyurl.com/tag-regulation-2020, 2020, [Online; Accessed 3 September 2021].
[2] ——, “How the TGA regulates software-based medical devices,”
https://tinyurl.com/tga-samd2021, 2021, [Online; Accessed 3 September2021].
[3] ——, “Medical device cyber security information for users,”
https://tinyurl.com/tga-md-cs2019, 2019, [Online; Accessed 3 September 2021].
[4] Zimmeck, Wang et al., “Automated analysis of privacy requirements for
mobile apps,” in NDSS, 2017.
[5] Wyatt, “How can clinicians, specialty societies and others evaluate and
improve the quality of apps for patient use?” BMC medicine, vol. 16,
no. 1, pp. 1–10, 2018.
[6] Australian Digital Health Agency, “Safe, seamless and secure: Evolving health and care to meet the needs of modern Australia,”
https://tinyurl.com/adha-plan-2017, 2020, [Online; Accessed 3 September 2021].
[7] Office of the Australian Information Commissioner, “Guide to securing personal information,” https://tinyurl.com/oaic-2018, 2018, [Online;
Accessed 3 September 2021].
[8] ——, “Guide to health privacy,” https://tinyurl.com/oaic-2019, 2019,
[Online; Accessed 3 September 2021].
[9] Kasperbauer and Wright, “Expanded FDA regulation of health and
wellness apps,” Bioethics, vol. 34, no. 3, pp. 235–241, 2020.
[10] European Commission, “Privacy Code of Conduct on mobile health
apps,” https://tinyurl.com/privacy-code-conduct, 2020, [Online; Accessed 10 February 2021].
[11] Office of the Australian Information Commissioner, “Australian Privacy Principles,” https://tinyurl.com/oaic-app, 2014, [Online; Accessed
3 September 2021].
[12] ——, “Mobile privacy: A better practice guide for mobile app developers,” https://tinyurl.com/oaic-guide2014, 2014, [Online; Accessed 3
September 2021].
[13] Australian Commission on Safety and Quality in Health Care,
“National safety and quality digital mental health standards,”
https://tinyurl.com/safetyquality2020, 2020, [Online; Accessed 3
September 2021].
[14] Maheu, Nicolucci et al., “The interactive mobile app review toolkit
(IMART): A clinical practice-oriented system,” Journal of Technology
in Behavioral Science, vol. 1, no. 1, pp. 3–15, 2016.
[15] Torous, Chan et al., “A hierarchical framework for evaluation and
informed decision making regarding smartphone apps for clinical care,”
Psychiatric Services, vol. 69, no. 5, pp. 498–500, 2018.
[16] Camacho, Hoffman et al., “Technology evaluation and assessment
criteria for health apps (TEACH-Apps): Pilot study,” Journal of Medical
Internet Research, vol. 22, no. 8, p. e18346, 2020.
[17] Iwaya, Fischer-Hübner, Åhlfeldt, and Martucci, “Mobile health systems
for community-based primary care: Identifying controls and mitigating
privacy threats,” JMIR mHealth and uHealth, vol. 7, no. 3, p. e11642,
2019.
[18] Stoyanov, Hides et al., “Mobile app rating scale: A new tool for
assessing the quality of health mobile apps,” JMIR mHealth and uHealth,
vol. 3, no. 1, p. e27, 2015.
[19] Terhorst, Philippi et al., “Validation of the mobile application rating
scale (MARS),” Plos one, vol. 15, no. 11, p. e0241480, 2020.
[20] Llorens-Vernet and Miró, “Standards for mobile health–related apps:
Systematic review and development of a guide,” JMIR mHealth and
uHealth, vol. 8, no. 3, p. e13057, 2020.
[21] Levine, Co et al., “Design and testing of a mobile health application
rating tool,” NPJ Digital Medicine, vol. 3, no. 1, pp. 1–7, 2020.
[22] Leigh, Ouyang, and Mimnagh, “Effective? Engaging? Secure? Applying
the ORCHA-24 framework to evaluate apps for chronic insomnia
disorder,” Evidence-based mental health, vol. 20, no. 4, pp. e20–e20,
2017.
[23] International Medical Device Regulators Forum (IMDRF),
“Principles and practices for medical device cybersecurity,”
https://tinyurl.com/imdrf-2019, 2020, [Online; Accessed 3 September
2021].
[24] Australia’s National Digital Health Initiative, “Digital Health:
The sleeping giant of Australia’s health technology industry,”
https://tinyurl.com/andhealth-2020, 2020, [Online; Accessed 16 February 2021].
[25] Baumel, Faber, Mathur, Kane, and Muench, “Enlight: A comprehensive
quality and therapeutic potential evaluation tool for mobile and webbased eHealth interventions,” Journal of Medical Internet Research,
vol. 19, no. 3, p. e82, Mar 2017.
[26] Benjumea, Ropero, Rivera-Romero, Dorronzoro-Zubiete, and Carrasco,
“Privacy assessment in mobile health apps: Scoping review,” JMIR
mHealth and uHealth, vol. 8, no. 7, p. e18868, 2020.
[27] Bondaronek, Alkhaldi, Slee, Hamilton, and Murray, “Quality of publicly
available physical activity apps: Review and content analysis,” JMIR
mHealth and uHealth, vol. 6, no. 3, p. e53, Mar 2018.
[28] Huckvale, Prieto, Tilney, Benghozi, and Car, “Unaddressed privacy risks
in accredited health and wellness apps: A cross-sectional systematic
assessment,” BMC Medicine, vol. 13, no. 1, Sep 2015.
[29] Huckvale, Torous, and Larsen, “Assessment of the data sharing and
privacy practices of smartphone apps for depression and smoking
cessation,” JAMA Network Open, vol. 2, no. 4, p. e192542, Apr 2019.
[30] Hutton, Price et al., “Assessing the privacy of mHealth apps for selftracking: Heuristic evaluation approach,” JMIR mHealth and uHealth,
vol. 6, no. 10, p. e185, Oct 2018.
[31] Karekla, Kasinopoulos et al., “Best practices and recommendations for
digital interventions to improve engagement and adherence in chronic
illness sufferers,” European Psychologist, vol. 24, no. 1, pp. 49–67, Jan
2019.
[32] O'Loughlin, Neary, Adkins, and Schueller, “Reviewing the data security
and privacy policies of mobile apps for depression,” Internet Interventions, vol. 15, pp. 110–115, Mar 2019.
[33] Scott, Richards, and Adhikari, “A review and comparative analysis of
security risks and safety measures of mobile health apps,” Australasian
Journal of Information Systems, vol. 19, Nov 2015.
[34] Australian Government Department of Health, Australian Digital Health
Agency (ADHA), “Cloud services: Considerations for healthcare organisation,” https://tinyurl.com/dh-cloud-2020, 2020, [Online; Accessed 3
September 2021].
[35] ——, “Selecting secure IT products and services: Questions to ask
your IT vendors. A companion for the information security guide
for small healthcare businesses,” https://tinyurl.com/adha-202, 2020,
[Online; Accessed 3 September 2021].
151
Authorized licensed use limited to: Nazarbayev University. Downloaded on April 10,2023 at 05:25:27 UTC from IEEE Xplore. Restrictions apply.
TABLE A1: ACSIMA Criteria List
Stakeholder Group
# Criterion
Authorized licensed use limited to: Nazarbayev University. Downloaded on April 10,2023 at 05:25:27 UTC from IEEE Xplore. Restrictions apply.
PM
C1. The app is developed by a trusted developer.
3
C2. The app has an Australian sponsor to be accountable for its quality, safety or performance in case the app is not Australian.
3
C3. The app declares whether it has been subject to a regulatory scrutiny or not.
3
C4. The app takes reasonable steps to ensure compliance with Australian applicable legislation, including: (1) Privacy Act 1988; (2) State and Territory Privacy Legislation; and (3) My health record
3
legislation.
C5. The app has the ability to meet digital health regulation in major markets overseas.
3
C6. The app has a complete and detailed privacy policy.
3
C7. The app explains the security systems used, including documented security risks and mitigations.
3
C8. The app has been tested by potential end-users before being made available to the public.
3
C9. The app declares the data to be collected including personal identifiable information (PII) and purpose(s) for collection.
3
C10. The app ensures the right of easy access to collected information.
3
C11. The app gives detailed information about any kind of agreements with third parties.
3
C12. The app requires users to always give their explicit consent before any action is taken.
3
C13. The app instantly notifies users of any cyber security risks when they become known.
3
C14. The app guarantees secure collection, storage and exchange of information.
3
C15. The app declares who owns the data that is collected.
3
C16. The app declares whether it allows the users: (1) to opt out of data collection or not; (2) to delete previously collected data or not; and (3) to share the data collected with other entities or not.
3
C17. The app developer has a data breach response plan that includes documented processes for taking appropriate action if a suspected or actual data breach occurs.
3
C18. The app guarantees that in case it lacks sufficient fundamental or inherent protective measures, and updates are not feasible, risk-mitigating alternatives will be applied as compensating controls.
3
C19. The app developer guarantees that any vulnerability will be corrected and disclosed in all jurisdictions in which the app operates.
3
C20. The app developer always coordinates with the regulatory authority to determine if a software change requires approval prior to release.
3
C21. The app provides contact information for end-users to report cyber security incidents and events, or at least the users can still submit through regular customer support channels.
3
C22. The app developer guarantees that cyber security incidents which impact user safety and privacy will be reported to applicable regulatory agencies as required by regulation. When a criminal activity is 5
identified through the course of investigation, local and applicable law enforcement agencies will be notified.
C23. The app uses the data collected only for the purpose that is explicitly declared.
5
C24. The app gives users control over their data.
5
C25. The app guarantees the availability of security updates when needed, and states if an update modifies or affects how it will function.
5
C26. The app developer provides basic cyber security training through the app to create security awareness and introduce cyber hygiene practices among all users.
5
C27. The app developer guarantees that healthcare providers have access to any information needed to implement a recommendation, and to ensure the protection of their patients.
5
C28. The app declares potential software-related risks and states the risk classification (if known). If the app is classified as high-risk, it has to additionally declare whether it was subject to third-party
5
oversight or not.
C29. At the verification and validation stage in the design and development process of the app, various types of security testing were employed to provide assurance that the code is free of significant
5
known vulnerabilities and that security controls have been effectively implemented.
C30. The app guarantees periodic security testing that can include: static code analysis, dynamic analysis, robustness testing, vulnerability scanning and software composition analysis.
5
C31. The app identifies cyber security vulnerabilities, detects possible threats, and applies security measures to minimise the risks.
5
C32. The app implements an encryption mechanism to secure collection, storage and exchange of information.
5
C33. The app implements reliable authorisation and authentication mechanisms.
5
C34. The app secures data transfer to prevent unauthorised access, modification, or replay.
5
C35. The app declares if confidentiality risk control measures are required to protect message control/sequencing fields in communication protocols or to prevent the compromise of cryptographic keying
5
materials.
C36. If the app is open source, the main developers declare how it will be updated and controlled.
5
C37. The app implements mechanisms to detect, resist, respond and recover from cyber security attacks in order to maintain its essential performance.
5
C38. Documentation is available that shows the app architecture, and where the data will be stored (e.g., user information is stored within Australia, to meet privacy requirements).
5
C39. The app is designed and developed under a secure development framework to minimize the number of legacy components in the future, and at a minimum, it meets a security baseline and includes
5
mechanisms for updates and patches.
C40. The app integrates a cyber security risk management process that: (1) identifies any cyber security vulnerability; (2) estimates and evaluates the associated risks; (3) controls those risks to an acceptable 5
level; (4) assesses and monitors the effectiveness of the risk controls; and (5) communicates risks via coordinated disclosure to key stakeholders.
C41. The app declares whether it stores data on the device, on the cloud or on both.
5
C42. The app explicitly states terms and conditions of the used cloud service, and makes sure that relevant security measures are present.
5
C43. Every time an update of a third-party component is published, the changes are inspected, and the risk is revaluated.
5
C44. As cyber security threats will continuously evolve, the app proactively monitors, identifies and addresses vulnerabilities and exploits as part of its cyber security management plan across the total
5
product life cycle.
C45. The app implements a formalized process for gathering information from vulnerability finders, developing mitigation and remediation strategies, and disclosing the existence of vulnerabilities and
5
mitigation or remediation approaches to stakeholders.
C46. The app developer documents and summarises their activities related to cyber security including test reports that describe all tests performed to verify the security of the app and the effectiveness of
5
any security controls.
C47. The app developer uses plain language, at an appropriate reading level for the intended user, to communicate actionable information regarding cyber security vulnerabilities and threats.
5
C48. The app allows anonymised sharing of information to improve security and user safety. The app explicitly declares that the shared information is not to be used to gain a commercial advantage.
5
C49. The app applies appropriate measures to protect minor users in accordance with applicable legislations.
5
C50. The app developer guarantees to continuously inform consumers, healthcare providers and regulators about any problem in released software versions and how to obtain and install changes.
5
152
N
HP
R
SE/AD C
O
3
3
5
3
5
5
5
3
5
5
5
3
3
3
5
3
3
3
5
3
4/6
4/6
1/6
6/6
5
3
5
3
5
3
5
5
5
3
3
3
3
5
3
5
3
5
3
3
3
3
3
3
5
3
3
5
3
3
3
5
5
5
3
3
3
3
3
3
3
3
3
3
3
5
3
3
3
5
3
5
3
3
5
3
3
3
3
3
3
3
3
5
3
3
3
5
5
5
3
5
5
3
3
3
3
3
3
3
3
5
3
3
3
5
5
5
3
5
3/6
6/6
5/6
6/6
5/6
6/6
4/6
5/6
5/6
2/6
6/6
6/6
6/6
1/6
3/6
1/6
6/6
2/6
3
3
3
3
3
5
3
5
3
3
5
3
5
5
3
5
5
3
3
3
3
3
5
3
3
3
3
3
5
3
4/6
3/6
5/6
4/6
1/6
4/6
5
3
5
5
5
1/6
5
5
5
5
5
5
3
3
3
3
3
3
3
3
3
3
5
5
5
5
5
5
5
5
5
5
5
5
5
5
2/6
2/6
2/6
2/6
1/6
1/6
5
5
5
5
3
3
3
3
3
5
5
3
5
5
5
5
5
5
5
5
2/6
1/6
1/6
2/6
5
5
3
5
5
1/6
5
5
5
5
5
5
5
5
3
3
3
3
5
5
5
5
5
5
5
5
1/6
1/6
1/6
1/6
5
5
3
5
5
1/6
5
5
3
5
5
1/6
5
5
5
5
5
5
5
5
3
3
3
3
3
5
3
3
3
5
3
3
3/6
1/6
3/6
3/6