IS YOUR HEALTH SURVEY
RESEARCH AS SMART AS
YOUR PHONE?
6/14/2011
Trent D. Buskirk, Ph.D.
Saint Louis University, School of Public Health
PART I: THE CURRENT
TELEPHONE/SMARTPHONE
LANDSCAPE…
The opportunities for innovation in Health
Surveys…
Putting the Pieces Together…
85% of U.S.
Households own at
least one Mobile Phone
Blumberg & Luke,
2011
87.4% of U.S. Adults
own a Mobile Phone
Blumberg & Luke,
2011
23% of Americans are
Cell Phone Only
27.8% of U.S. Adults
live in Cell Only HHs
Pew Internet, 2010
and Blumberg &
Luke, 2011b
37% of Cell Phone
Consumers have a
Smartphone
Nielsen Wire, 2011
3
Smartphone Market Share Estimates
from February-April 2011 (Nielsen, 2011)
http://blog.nielsen.com/nielsenwire/consumer/android-leads-u-s-in-smartphone-market-share-and-data-usage/
Public Health Context: Software

Smartphone prevention and treatment related apps
continue to be developed and deployed in Public
Health settings
Harvard’s iPhone Swine Flu Tracking app (Rao, 2009)
 Apps to improve patient/provider communications
(Patrick, et al. 2008)
 Wellness diary apps for health promotion (Koskinen and
Salminen, 2007)
 Diabetes maintenance and smoking cessation apps
(Logan et al., 2007 and Abroms, et al., 2011)

Public Health Context: Hardware

Smartphone prevention and treatment related apps
continue to evolve to include hardware peripherals
that can be used to extend the smartphone’s
capabilities…
Wahoo’s Fitness Packs
ThinkLab’s Electronic Stethoscope
(and Heart rate Monitoring App)
Focusing in on iPhones…






Majority of iPhone and Google Android Smartphone users report
being drawn to the devices because of diversity of apps available
(Helmreich and Dorit, 2009).
The number of iPhone apps now exceeds 134K
(newmaconline.com, 2010).
iPhones estimated to be the most common smartphone used to
access internet based surveys via mobile devices (Kinesis, 2010).
Roughly 40% of iPhone users are between 35 and 54 years old
(NeilsenWire, 2009).
Utilization of data services among 30-49 year old cellular phone
/Smartphone owners is continuing to rise (Smith, 2010).
An estimated 61% of Physicians use iPhones currently (Dolan,
2011).
Key Points for Health Surveys…



Health Surveys targeting prevention or risk related
activities among minority groups or key age groups…
Inquiries into the nature and use of specific types of
apps for health promotion, disease management and
prevention…
Smartphones equipped with peripherals give
researchers new ways to automate data collection
and potentially reduce:
recall bias
 noncompliance/item nonresponse
 overall measurement error

PART II: HEALTH SURVEYS
WITH SMARTPHONES…
It’s Not Just Another Online Survey…
Design Configurations…

Smartphone screen sizes vary in size, but this
form factor has implications for questionnaire
design including:
 Question
layout – require scrolling or not?
 Number of questions per screen
 Using/allowing page reloading (open- web like) or
not (app like)
 Drop down menus vs. free response data capture
 Use of icons/graphic images within the survey?
Design Considerations

Placement of Next and Back buttons
 Couper,

Data entry types/fields
 Couper

et al. (2011)
(2010) and Peytchev and Hill (2010)
Icon size, number per screen and placement
 Callegaro


(2010) and Peytchev and Hill (2010)
Splash page redirects
Native user experience (minimize “user anxiety”)
Design Considerations, Cont.


Couper’s 2011 experiment tested back and next
buttons displayed at the bottom of screen in
various combinations
In the iPhone mobile browser context,
 Back
buttons at the bottom of the screen will be
masked during data entry
 Placing Back and Next buttons above one another on
same side of screen may be difficult for some touch
screen users
Screenshots from our Study Illustrating
Back Button Placement
Dropdown Field
PART III: A FIELDED
SMARTPHONE SURVEY
FOCUSING ON HEALTH
RELATED APP USE…
With Specific Emphasis on Comparing
Computer and Smartphone Modes…
An App a Day Could Keep the
Doctor Away –Quantifying the
Use of Prevention Related
Smartphone Apps Among
iPhone Users
Trent D. Buskirk, Ph.D.
Charles Andrus, B.S.
Mark Gaynor, Ph.D.
Chris Gorrell, B.S.
Saint Louis University
School of Public Health
Primary Study Objectives
Evaluate mode effects across device in the context of iPhone general and health app use
Determine whether iPhone users recognize popular apps by name and/or icon.
Determine the extent of ownership and use of popular prevention apps
Correlate health behavior and outcomes with iPhone app use
Sample Description
iPhone users were recruited for this study using Survey Sampling Inc. International’s (SSI) Dynamix modern recruiting tool
Panelists from the online U.S. SurveySpot Panel were screened for type of cell phone. Random subset of qualified panelists were selected for further screening questions.
Final Phase Recruitment and screening occurred between May 2, 2011 and May 4, 2011.
Incentive for completing:
iPhone assignees: 400 points ($4.00) Computer assignees: 200 points ($2.00)
Sample
Allocation/Randomization
Prior to randomization to mode, eligible panelists were stratified by:
Education Level: <Bachelors and ≥ Bachelors
Age Group: <40 and ≥ 40
Sex
Randomization to mode was carried out separately within each of the 8 strata (approximately 75% to iPhone and 25% to Computer)
Research Questions:
Recruitment Oriented Outcomes
Are survey redirect rates higher for iPhone assignees compared to Computer assignees?
To what extent would iPhone assignees use the Text Message option? Would iPhone assignees have errors entering the survey’s web address? Are there differences in total survey time across mode?
Are there differences in the completion rates by mode?
Are there differences in the total number of typed characters reporting the names of “other” prevention apps across mode?
Design Considerations
Some guiding principles for our design process:
Maximum Number of Questions per screen:
4 per web screen
2 per iPhone screen
Answer choices on iPhone screen require as little screening as possible
in most cases the screen landscape suggested/ implied a need for scrolling
Design Considerations, Cont.
We attempted to provide “native” app‐like processing for the iPhone survey version by including:
“loading” pinwheel
“asynchronous Javascript XML” for improved screen transitions
App icons rendered as 72‐by‐72 pixel images Used “badges” to denote user selection of icons whenever apps were displayed as part of a question…
Secure streamlined web address that excluded special characters
Design Considerations - Illustrated
Implied Scrolling
Loading Pinwheel
72‐by‐72 pixel icons with badges
Which if any of the following apps have
you downloaded to manage your weight?
(Click all that apply)
Web‐based portion
Apps grouped together on iPhone‐ required scrolling
iPhone Invitation Process
All panelists initially invited on their lap/desktops
iPhone panelists were asked to point their iPhone
web browser to:
http://mobilehealth.slu.edu/1234567890 B
Secure Server Web Address
MODE Indicator
10‐digit Panel ID iPhone Panelists who followed the invitation link on their lap/desk top redirected to a “splash” page
An option to receive the link via SMS/text message was available
iPhone Redirect Splash Page
Survey Screener and
Response Flow
Panelists who Reported Owning iPhone: 2053
Online Panelists Screened for iPhone: 16051
Final Status
iPhone
Computer
Never Entered Cite
650
12
99
221
16
20
83
209
Ineligible
Partial Complete
Complete
Panelists Selected to receive Stratification Demographic Questions: 1339
# Panelists Randomized to Survey Mode: 1310
Computer: 328 iPhone: 982
Modeling Completion Rates
Wald Chi‐
Square
df
Sig.
STRATUM
32.636
7
<.0001
MODE (Computer)
24.545
1
<.0001
STRATUM * MODE 7.361
7
0.392
Constant
25.82
1
<.0001
Variable(s) entered: STRATUM, MODE, STRATUM * MODE Controlling for Stratum Assignment:
Computer Completion Rate ≈ 2.6 * iPhone Completion Rate
Process Results: Web Address Entry
Among iPhone Redirects…
We had a total of 12 iPhone completes (5.4%) that entered their Panel‐ids into the web address incorrectly (off by 1 digit)
http://mobilehealth.slu.edu/1234567890B
Examples:
9 Digit #s
10 Digit #
Entered ID
119039274
111848307
1079360247
ACTUAL ID
Process Results:
iPhone Completes…
We had a total of 12 iPhone completes (5.4%) that entered their Panel‐ids into the web address incorrectly
http://mobilehealth.slu.edu/1234567890B
Examples:
ENTERED ID
ACTUAL ID
_119039274
1119039274
1118484307
1079350247
111848_307
1079360247
Process Results: SMS and Redirects
Text Message Requests
15 of 332 (4.5%) iPhone panelists who visited the website requested SMS Redirects Among those who visited study site
5.8% of Computer Assignees
30.1% of iPhone Assignees (Fischer’s Exact p‐value <.00001)
Questionnaire Completion Rates
by Mode…
x  99.69%
x  99.58%
Loading Times…
LOADING TIME
S
C
R
O
L
L
Loading Times By Mode for
Question 1: Screen In
x  402.05
(316.25, 487.85)
n  258
x  449.17
(357.25,541.10)
n  241
Loading Time to Question 1 (in Milliseconds)
Nonparametric Density Estimates of Total
Survey Time in LN(Minutes) Among
Survey Completers
Total Number of Characters Entered
for Names of “Other Apps” By Mode
PART IV: THE TAKE-AWAY
POINTS…
What’s Smart about Smartphone Surveys and
what isn’t?
Final Thoughts…



As suspected online panelists assigned to complete
via iPhone have a lower completion rate overall
compared to those assigned to complete via
computer.
 Put another way- more iPhone assignees were
required (in about a three to one ratio) to have
approximately the same number of completes by
mode
Smartphone Surveys may require additional
programming to support app-like experiences within
the survey setting
Smartphone surveys may be appropriate for “hard to
reach or specialized populations” – another mode…
More Final Thoughts…



Smartphone users may take less time to complete
the survey compared to ordinary online
surveys…
Smartphone users may have more interruptions
or break-offs during the survey process –may
consider “next and save” versus final “submit
all” options for user.
Smartphone surveys can also be used by
interviewers in the field as an extension of the
“PDA” computer assisted interviewing
References








Abroms, L.C. and Padmanabhan, N. et al. (2011) iPhone Apps for Smoking Cessation: A Content Analysis Am
J Prev Med; 40 (3) 279–285
AAPOR Cell Phone Task Force Report, 2010; Retrieved from
http://www.aapor.org/Cell_Phone_Task_Force_Report.htm, accessed on February 20, 2011.
Blumberg SJ, Luke JV. Wireless substitution: Early release of estimates from the National Health Interview
Survey, July-December 2010. National Center for Health Statistics. June 2011. Available from:
http://www.cdc.gov/nchs/nhis.htm.
Coderre, F., Mathieu, A. & St-Laurent, N. (2004) Comparison of the quality of qualitative data obtained
through telephone, postal and email surveys. International Journal of Market Research, 46, 3, pp. 347–357.
Callegaro, Mario. 2010. “Do You Know Which Device Your Respondent Has Used to Take Your Online
Survey?” Survey Practice, December: www.surveypractice.org.
Cazes, J., Townsend, L., Rios, H., & Ehler-James, J. (2010). The mobile survey landscape – Today and
Tomorrow. Impacts of mobile devices usage on current and future market research practices. Retrieved from
http://www.kinesissurvey.com/files/MobileSurveyLandscape_KinesisWhitepaper.pdf
Couper, Mick P., Reg Baker, and Joanne Mechling. 2011. “Placement and Design of Navigation Buttons in Web
Surveys” Survey Practice, February: http://surveypractice.org
Couper, M. P. (2010). Visual design in online surveys: Learning for the mobile world. Presented at the Mobile
Research Conference 2010, London. Retrieved from
http://www.mobileresearchconference.com/uploads/files/MRC2010_Couper_Keynote.pdf
References, Cont.










Nurss JR, El-Kebbi IM, Gallina DL, et al. (1997) Diabetes in urban African Americans: functional health
literacy of municipal hospital outpatients with diabetes. Diabetes Educ; 23:563–8.
Okazaki, S. (2007). Assessing mobile-based online surveys. International Journal of Market Research, 49,
651-675.
Peytchev, A., & Hill, C. A. (2010). Experiments in mobile web survey design: Similarities to other modes
and unique considerations. Social Science Computer Review, 28, 319-335.
Sax, L. et al. (2003), “ASSESSING RESPONSE RATES AND NONRESPONSE BIAS IN WEB AND PAPER
SURVEYS,” Research in Higher Education, Vol. 44, No. 4, pp. 409-432
Smith, A. (2010). Mobile access 2010. Pew Internet & the American Life Project. Retrieved from
http://pewinternet.org/Reports/2010/Mobile-Access-2010.aspx
Smith, W.G. (2008) “Does Gender Influence Online Survey Participation? A Record-Linkage Analysis of
University Faculty Online Survey Response Behavior” ERIC database paper number ED501717,
accessed from
http://www.eric.ed.gov/ERICWebPortal/search/detailmini.jsp?_nfpb=true&_&ERICExtSearch_SearchValu
e_0=ED501717&ERICExtSearch_SearchType_0=no&accno=ED501717 , retrieved on May 7, 2011.
Townsend, L. (2005). The status of wireless survey solutions: The emerging “Power of the Thumb”.
Journal of Interactive Advertising, 6, 40-45.
U.S. Preventive Services Task Force (2009), “Aspin for the Prevention of Cardiovascular Disease,”
retrieved from http://www.uspreventiveservicestaskforce.org/uspstf/uspsasmi.htm, accessed on May 7,
2011.
Vicente, P., Reis, E., & Santos, M. (2009). Using mobile phones for survey research. International Journal of
Market Research, 51, 613-633.
Williams MV, Parker RM, Baker DW, et al. (1995) Inadequate functional health literacy among patients at
two public hospitals. JAMA 1995;274:1677–720.
The End!
Click Below to Exit!
Thank
You!
Process Related Results
Technical Flow for The Survey
Adminstration
Masked URL
https://mobilehealth.slu.edu/1A
https://mobilehealth.slu.edu/1B
MySQL
Ext JS
Touch
Geographic Distribution of Survey
Respondents
Geographic Distribution of iPhone Panelists
Randomized to Mode for This Study
Survey Completion Rates By
Assignment Stratum and Mode
The nature of the Stratum Effect…
Cumulative Survey Intake by
Study Collection Day
Next Steps/Conclusions
Text link requests were used by a small number of iPhone assignees
Redirect rates were significantly higher among iPhone assignees compared to computer assignees Loading Times…
U.S. mobile phone penetration as of June 2010


100
85.0% of U.S. Households owned a mobile phone
87.4% of U.S. Adults owned a mobile phone (Blumberg & Luke, 2011)
Mobile phone penetration in US - 2006-2010
Adults
Households
90
Percentage of
87.4
85.0
80
70
60
50
May-02
Nov-02
May
06 Nov
06
May-03
May
07
Nov-03
Nov
07
May-04
May
08
Nov-04
Nov
08
May-05
May
09
52
Nov-05
Nov
09
May-06
May
10
Percent of Adults by Age Who Live in CellOnly HHs, Over Time (Source: Blumberg & Luke, 2011b)
Mobile Phone Internet Traffic as Measured by
Within-Web Page Advertisements by Type of
Smartphone (Source: AdMob, 2010)
Race/Ethnicity Smartphone vs. Feature Phone…
(Source: Nielsen Wire, 2010)
http://blog.nielsen.com/nielsenwire/online_mobile/mobile-snapshot-smartphones-now-28-of-u-s-cellphone-market/
Penetration of Smartphones versus Regular
Cell Phones by Age-group (Source: ComScore Data, 2010)
http://www.comscoredatamine.com/2010/09/u-s-smartphone-users-by-age/