GALINA MAKAGONOVA
GADGET ON THE BUDGET
How to budget, create and operate a
Device Test Lab
Average size of mobile development team is 20-70 people
GAMES
MOBILE APPS
MOBILE WEB
Our Agenda
1. WORKSHOP
Device lab requirements
Device parameters
Planning: location, equipment, software, storage
Budget
Device lab technician: role description and responsibilities
Where to buy devices and overview of issues
Lab Maintenance
Device usage in Agile development team
How to present device lab during negotiations with the clients
2. PRACTICE and Q&A
OPTIONS TO EXPLORE
OR NO DEVICE LAB?
NO DEVICE LAB
Application delivered
to client without
testing
Developers and testers are
using their own devices
Or Using device emulator
Or BYOD policy
External Test Lab
TYPES:
1. Device Rental Labs
2. Crowd solution (using private devices across the globe)
3. Free device lab
PRO: no need to invest in device acquisition
CONS:
1. Cost:$15-25 per device per usage, $25-$30 per hour device+ tester
2. Confidentiality: your app might be exposed to the market
3. Timeline: Sending out your testers at the end of test cycle
4. No flexibility in OS update/Version
5. Your target devices may not be available
6. Booking in advance
7. No way to reproduce bug on device
8. Developers cannot investigate the bugs and validate features
Overview: Possible roadblocks
Do You Need Internal Test Lab
1. What kind of mobile application are you developing?
Mobile Web? Games? Mobile apps?
2. How many projects do you have at the same time?
3. How many developers are in the company?
4. Are development and QA co-located?
5. Do you have a test department? How many testers?
6. Are you developing the app for Android/iOS market? Or for your
clients?
7. Do you have contract obligations to test the app on mobile
devices?
8. Agile or Waterfall?
9. When is Device test lab usage required? At the end of development
cycle or for everyday usage?
Overview: various roadblocks
Business Focus
1) Who are the users?
2) Who will pay (how to negotiate with the clients or get management
buy-in)?
3) How many devices do you need or could you afford?
4) Device selection/Device matrix
5) Device fragmentation (specifically, Android fragmentation)
6) Managing device inventory
7) What is the budget?
8) Device selection should be relevant for a year, preferably for 2
9) Return of investment: what is the value of your lab?
Who Are The Internal Users Of A Mobile Test
Lab?
1. Development teams: How many?
2. Developers – how many?
3. Testers
4. Designers
5. Artists
6. Customer support / Publishing
7. Sales
8. User acceptance coordinators
Who: Target audience
Where to get info: from GD, Sales, Data analytics, GOOGLE
1. Market: countries
2. How secure the application is? Do you need to test hackers attacks?
3. Age and gender of the users
Overview: various roadblocks
And: Technical requirements
Where to get info: from GD, Lead developers, game teams
1. IOS and Android? Android only? iOS only?
2. Any specific hardware requirements (i.e. gyroscope)
3. Screen requirements
4. Memory requirements
5. Graphic processor requirements
6. Other specific requirements
7. Phones and tablets? Or just phones? Or only tablets?
8. Pre-installed applications
9. Application to download
Overview: various roadblocks
Device Parameters:
•
•
•
•
•
•
•
•
CPU
GPU
RESOLUTION
CHIPSET
SENSORS
YEAR RELEASED
PRICE
MARKET RELEASED
Device Lab Planning
Centralized or Distributed?
Static lab: Round table for testing (devices are co-located, anyone who
want to test should come there.
Agile world: Devices are assigned per team and per tester, with visibility
for the whole team: anyone who needs it could borrow for 1 hour or
so.
•
•
•
•
•
•
•
•
Location
Storage Space
Tracking tool: I’m using Meraki
Cables, cleaning, power supply
Inventory
Bookkeeping/Accounting
Internal Policy
Disasters happen
Overview: Common mistakes
Budget
• List of devices
• Cost of devices
• Cost of tools and supplies
• Cost of device lab (storage, etc)
• Cost approval process
• Lab technician (salary)
• Who will pay
Overview: typical mistakes
More About Device Lab Budget
• Per project
• Per tester
• Track device life
• Maintenance materials
• Plastic gloves
• Cables
• Power supply
• Charging stations
• Tape for marking cables and devices
• SIM cards
• Budget for testing mobile data
Device Lab Technician
Keep inventory changes
Track lost devices
Plan and manage OS updates
Root devices if needed
Sideload client build
Charge all devices
Gather devices spread across office
Create device test list per app
Perform device testing per app
Perform compliance testing
What To Buy And How Many
List devices to cover the technical spec (low end, high end, phones and tablets)
How many teams do you have?
How many developers per team? Total?
How many testers per team? Total?
Each QA should have 1 Android and 1 IOS gadget (1 phone, 1 tablet)
Each team should have 1 high end, 1 low end (both iOS and Android)
Who managed provisioning profile? How many devices are already in?
Some samples of requests to address
Buying devices
*** No contract
*** New or used
*** Beware of jailbroken devices
• Amazon
• E-bay
• Classifieds
• Cell phone providers
• Buying old devices from staff (how much do you want to pay)
Lab Maintenance
• Inventory (locate each gadget bi-monthly)
• Mark each device, cable and charger
• Cable replacement
• Monitor chargers
• Buy sleeves
• Monitor upcoming OS changes
• Monitor new devices on the market
• Manage device assignments to team members
Negotiation
Presenting options with cost and risk
Cost of device rent per test cycle, availability
Outsource – cost per cycle
Developers and QA need devices during development cycle
In-house, including
Cost of device tester
Quantity of test cycles
List of devices per app requirements
*** Who will own devices at the end of development
*** Add 1/6 of device budget every 6 months to keep current
Workshop
• List of 30 Devices with larger market share (samples)
• Operation system diversity
• Practical exercise: reviewing application requirements and
selecting 10 top devices
• Practical exercise: Working on device lab budget
• Practical exercise: Managing device test lab
• Practical exercise: Device compatibility checklist
Thank you very much for attending!
See you soon for the hands-on Training.
Galina Makagonova
QA Director of Roadhouse Interactions,
Vancouver, Canada
Galina Makagonova
Gadget on the budget