UI Automation Tools
Ranorex
Appium
Calabash
Tool Evaluation
Ranorex - Advantages
●
● Cross platform support
● Test Automation Library for .Net
● Standard programming langauges C# and VB.Net for scripting
● Test development environment (Ranorex Studio) features
recorder, debugger, object spy and repository
● Flexible Test Automation Library Interface Test suites with
Ranorex results in .EXE files for simple integration with test
management, behaviour driven development and continuous
integration server tools
● Customizable Test report engine.
How does it work?
Gherkin
4. Specflow reads bdd
(formatted in Gherkin)
3. Team city runs Visual
Studio project
Run Tests
(TeamCity Server)
SpecFlow
Ranorex
+
Library
Visual Studio
2. .Exe is
stored in
repository
SVN
3333
Device/Simulator
5. Connects to
device through
WiFi
6. Test Scripts
runs on RT App.
Android
RT App
(Our Build)
iOS
1.BDD Stories are turned
into Visual studio projects
Windows
BDD Stories
Ranorex-Disadvantages
●
● Xamarin related automation Issues
● Support takes more time for Xamarin related issues
● Ranorex agent compatibility with Application is poor and
causes crashing as new features are introduced in app
● Instrumented application is slower than original Application
● Some Touch gestures/Drag and drop operation not supported
● In case of crashing/hanging, Automatic launching of closed
Application is not supported
● Need to provide long waiting times for object identification.
Appium - Advantages
• Active community
• No extra code into application
• Open source – we can make our own fixes and
ask for pull requests on github
• Easier to learn and implement than Calabash
• Element inspector in Appium app
Appium - Disadvantages
• Over 50 open bugs with iOS
• Unable to get hooks on certain elements in
complicated apps(programme detail page).
• https://github.com/appium/appium/issues/537
• Inconsistent with outcomes(due to no
wait/delay)
• Currently cannot drag – known logged bug
Appium - Demo
Calabash - Advantages
•
•
•
•
Now owned by Xamarin
Wait/sleep functionality
Can swipe
BDD Layer easily configurable
Calabash - Disadvantages
• Not tested enough?
• No updates in months – maybe due to takeover
from xamarin so may become closed source.
• Must be written in Ruby and cucumber - Can
be tricky
Calabash - Demo
Conclusion
Best tool to use is Calabash:
● Owned and supported by Xamarin
● More stable and reliable, testing is more
consistent.
● Easier to setup the BDD layer for QA’s
How does it work?
Gherkin
1. Team city runs
Cucumber scripts.
Run Tests
(TeamCity
Server)
2. Cucumber reads BDD
(formatted in Gherkin)
Cucumber
3. From the parsed Gherkin
files, Cucumber runs the
retrieved Ruby files and
methods.
Ruby
Android
Calabash
iOS
WP
4. Ruby talks to calabash to
run commands on the
platforms.
Xamarin
(Our Build)