Mobile Testing
Karen N. Johnson
© Karen N. Johnson, 2012
1
who am I?
Karen N. Johnson
software test consultant based in
Chicago
website: www.karennjohnson.com
twitter: @karennjohnson
email: karen@karennjohnson.com
2
where are we?
the mobile world
3
Are you testing with mobile
browsers or a mobile app?
4
manufacturers, models, operating
systems, network carriers, operating
systems, firmware upgrades, mobile apps
& mobile websites and yet ...
so little time to test.
5
mobile manufacturers
Manufacturers change
continually.
Manufacturers vary by
country.
Understand the market
your app is in vs. the
overall global market.
Expect to update your
strategy frequently.
6
device models
Device models change
frequently as well.
Devices also vary by
country.
Device detection: do
you know what your
users have?
7
operating systems
Android from Google, Inc.
BlackBerry OS from RIM
iOS from Apple
Symbian OS from the
Symbian Foundation
WebOS from HP
Microsoft Windows Mobile
- Source Wikipedia
8
mobile browsers
Users can download a variety of browsers in addition to
the native browser on the device.
Android browser
BlackBerry browser
Safari
Internet Explorer
Firefox
Opera
Dolphin
9
network carriers
Connection is based on
country, carrier, signal
and the device.
Network carriers can
impact SMS texting
and access to
websites.
10
mobile statistics
Source: Which source
do you trust?
Country: Usage
statistics vary greatly
by country.
Analytics: Are you
capturing device usage
for your own
application?
11
find your mobile sanity
Understand the global mobile market.
Keep watching the market.
Know where your app is within the market.
Keep watch on what your competitors offer.
Realize what you can do and what you cannot do.
12
Quiz Time !
What is a walled garden?
or perhaps you refer to it as a
carrier deck?
13
walled garden/carrier deck
“...seasoned mobile developers suffer from carrier
(operator) myopia. That is to say, they believe carriers
(e.g. Vodafone, Orange, AT&T, Verizon, etc.) have, and
will continue to have, complete control over the
availability of web content through their networks.
Traditionally this has been the case, as most users had
restricted access only to websites approved by the
operator or carrier, often referred to as a carrier deck,
closed portal, or walled garden.”
Cameron Moll, Mobile Web Fundamentals
14
mobile-centric testing
UX: user experience
app permissions
connectivity
device settings
installation
mobile interruptions
user scenarios
SD Cards
OS & firmware updates
SIM Cards
15
UX
User interface considerations on mobile
are radically different than on the desktop.
16
connectivity
We are not stationary
when we use mobile.
So why are we testing
from the inside of our
quiet well-connected
offices with fully charged
batteries?
17
installation
Application install & updates combined with OS updates
combined with device or browser settings.
18
user
scenarios
The navigational
path users take
on mobile is not
the same as on a
desktop.
19
OS & firmware updates
20
app
permissions
Your app
asks for what?
21
device settings
user interface considerations on mobile
are radically different than on the desktop.
22
Quiz Time !
What does WURFL mean?
23
WURFL
WURFL = Wireless Universal Resource File
The WURFL is an "ambitious" configuration file that
contains info about all known Wireless devices on earth.
http://wurfl.sourceforge.net/backgroundinfo.php
http://www.scientiamobile.com
Mobile device detection, http://www.hand-interactive.com/resources/detectmobile-javascript.htm
Device Atlas, http://deviceatlas.com/
24
mobile interruptions
Forced updates
Incoming calls & texts
Battery issues
Connectivity issues
25
SD Cards
Does your app store
data on a card?
What if the card is
removed?
What if a different card
is inserted?
26
SIM cards
A SIM card or
Subscriber Identity
Module is a chip.
SIM holds personal
identity information, cell
number, phone book,
text messages and
other data.
Again, what happens if
the card is full?
Changed? Removed?
27
mobile test challenges
1. What devices should we test with?
2. We don’t have any analytics yet.
3. Screen shots are hard to get.
4. Replicating defects is not the same as it used to be.
5. The testing tool market is not there yet.
28
choosing what devices
to test
29
screen capture tools
WebCapture
Apple. DJM Development
http://itunes.apple.com/us/app/webcapture/id331452784?mt=8
Apple iPhone and iPod Touch
Hold the "Home" button, then press the "Power/Sleep" button. The
screenshot will automatically appear in your photo gallery.
Dolphin Screen Cut
Android. Dolphin Browser Add-on
https://market.android.com/details?id=mobi.mgeek.ScreenCut&hl=e
n
30
testing tools
TestFlight
FoneMonkey: iOS
Apple xCode: Mac, iPhone and iPad
Robotium: Android
eggPlant: automation tool to work across all devices
Perfecto Mobile
Device Anywhere
31
Quiz Time !
What is jailbreaking?
32
iOS jailbreaking
iOS jailbreaking, or simply
jailbreaking, is the process of
removing the limitations imposed by
Apple on devices running the iOS
operating system through use of
custom kernels.
- wikipedia
33
relevant “traditional” testing
functional
search
navigation
security
e-commerce (now m-commerce)
cookies
interface testing
34
functional testing
Here’s an example of a mobile app from UPS. Notice how
much traditional functional testing there is to do.
Track packages
Ship packages
Locate a store
Get a quote
Give feedback
35
search
Classic testing with a new twist: search with barcodes or
where device location is factored into the results.
36
navigation
We are still navigating sites and apps but
rarely do we have a visual cookie crumb trail.
Rotate
Pinch/zoom
Accelerometer
Vertical/horizontal
Paging forward & back
37
security
User interface considerations on mobile
are radically different than on the desktop.
38
m-commerce
The shopping cart with barcodes, location-based (NFC)
coupons, card scanning/swiping and Google wallet.
39
Quiz Time !
What is rooting?
40
rooting
Rooting is a process allowing users of mobile phones,
tablets PCs, and other devices running the Android
operating system to attain privileged control (known as
“root access.”) within Android's subsystem.
- wikipedia
41
cookies
Remember me?
Forgotten password
Auto-login
Third party login
access (i.e. Use
Facebook account to
login)
Password changed on
the desktop.
42
interface testing
Watch for apps working
together. Like Yelp!
and Open Table.
Consider appdependencies on
phone settings - like
location-based
searching.
43
where are we now?
develop a strategy
watch the market
watch competitors
watch “other” apps and
sites for innovation
develop mobile testing
skills & experiences
44
Q&A
Thank you for your time.
email: karen@karennjohnson.com
45
resources
mobile usage stats
http://mobithinking.com/mobile-marketing-tools/latest-mobile-stats
http://www.cloudfour.com/a-comprehensive-guide-to-mobile-statistics/
Google offers free web statistics. To use the service, you need to apply
and setup the service – from there you can track device statistics.
http://www.google.com/analytics/
mobile manufacturers, http://www.mobileforum.com/
mobile browsers,
http://webtrends.about.com/od/mobileweb20/tp/list_of_mobile_web_browse
rs.htm
mobile testing communities, http://www.mobileqazone.com/
46
resources
Product offerings to test mobile devices remotely.
http://www.deviceanywhere.com
http://www.perfectomobile.com
quick tests for mobile readiness
http://ready.mobi/launch.jsp?locale=en_EN
http://www.gomez.com/mobile-readiness-instant-test/
http://validator.w3.org/mobile/
http://ipadpeek.com/
47
resources
mobile manufacturers, http://www.mobileforum.com/
mobile browsers,
http://webtrends.about.com/od/mobileweb20/tp/list_of_m
obile_web_browsers.htm
A worldwide community with local chapters for mobile
design - Mobile Monday, known as MoMo.
http://www.mobilemonday.net/
48
jonathan kohl
Designing Mobile Apps: Using Old School Tools for New School
Technology Article, Modern Analyst, April 2011
Mobile Challenges for Project Management, Part 2: Human Factors Article,
Stickyminds, January 2011
Mobile Challenges for Project Management, Part 1: Project Factors Article,
Stickyminds, January 2011. Re-published January 2012 by Better Software
Magazine
Test Mobile Apps with I SLICED UP FUN! Article, self-published, October
2010. Republished December 2010 by Mobile Developer Magazine
49
julian harty
A Practical Guide to Testing Mobile Smartphone
Applications
http://www.mobileapptesting.com/julian-harty-mobileapp-testing-tutorial/2011/03/
http://www.ebaytechblog.com/2011/06/06/the-8thedition-of-mobile-developers-guide/
twitter: @julianharty
50
mobile articles
Karen N Johnson, “The Mobile Environment”
The Testing Planet, November 2011
Karen N Johnson, “What If” in Mobile Testing
Tea-Time with Testers, July 2011
Stephen Janaway,
http://www.thetestingplanet.com/2011/11/mobile-testingthat-s-just-a-smaller-screen-right/
Brad Frost, http://bradfrostweb.com/blog/mobile/test-onreal-mobile-devices-without-breaking-the-bank/
•
51
mobile books
Programming the Mobile Web, by Maximiliano Firtman
A Practical Guide to Testing Wireless Smartphone Applications, by Julian
Harty and Mahadev Satyanarayanan
Strategic Mobile Design: Creating Engaging Experiences, by Joseph
Cartman and Richard Ting
Mobile Design and Development, by Brian Fling
Mobile Web Design, by Cameron Moll
Designing Mobile Interfaces,by Steven Hoober and Eric Berkman
52
twitter
Luke Wroblewski, @lukew
Maximiliano Firtman,@firt
Cameron Moll, @cameronmoll
tomiahonen, @tomiahonen
Pete Cashmore, @mashable
@karennjohnson - see who I
follow
53