Automating Testing of Service-oriented Mobile
Applications with Distributed Knowledge and
Reasoning
SOCA 2011
December 14, 2011
James Raymon Edmondson
james.r.edmondson@vanderbilt.edu
Department of Electrical Engineering & Computer Science
Vanderbilt University, Nashville, TN, USA
Table of Contents
James R. Edmondson
Overview of Presentation Flow
•
•
•
•
•
•
Introduction
Motivating Scenario
Related Work
Solution Approach
Case Studies
Conclusion
Automating Testing with Knowledge and Reasoning
2
Introduction to DRE Systems
James R. Edmondson
Problem Overview
• DRE Systems
• Examples
– Large scale
– Less resources available than
adequate for all participants in
the system at full usage
• Network capacity
• CPU and memory
– Some participants are more
important than others
• Require differentiated service
during contentious periods
– Often mission critical
• Lives are often at stake
• Cannot go down or become
unresponsive
Automating Testing with Knowledge and Reasoning
–
–
–
–
–
–
–
–
–
Shipboard computing
Satellite software
Traffic light management
Ambulance services
Search and rescue missions
Conveyor belt systems
Avionics control systems
Missile guidance systems
Even public clouds could use
differentiated service with
their resources
3
Introduction to Automated Testing
James R. Edmondson
Testing DRE Systems
Automated Testing of DRE systems is difficult
– Heterogeneous platforms (hardware, OS and networking)
– Dynamic sequencing may require millisecond or less precision
– Testing hardware is often antiquated and failures are frequent
• Backups should be not only supported but encouraged in testing
– Large scale tests are often hard to visualize
Testing of mobile applications adds other issues
– Sequencing must be able to interact with smartphones
– How do we instrument the smartphone applications?
• Starting an application
• Interacting with the phone like a user would
Automating Testing with Knowledge and Reasoning
4
Motivating Scenario
James R. Edmondson
Motivating Scenario
network
•
•
Three computers and two smart phones connected to a network
The two smart phones are connected to two of the computers
Automating Testing with Knowledge and Reasoning
5
Motivating Scenario
James R. Edmondson
Motivating Scenario
server
System
Services
network
•
•
The phones communicate with system services on the server
The phones must wait for the system services to warm up before
connecting (5 seconds)
Automating Testing with Knowledge and Reasoning
6
Motivating Scenario
James R. Edmondson
Motivating Scenario
server
Phone
App 1
System
Services
network
Phone
App 2
•
•
A race condition exists if App 2 is started immediately after App 1
But it requires microseconds of precision to catch
Automating Testing with Knowledge and Reasoning
7
Motivating Scenario
James R. Edmondson
Motivating Scenario
server
Phone
App 1
System
Services
network
Process
Process
Results
Results
Phone
Phone
App 3
App 2
Process
Process
Results
Results
•
After the tests are performed, each host must process results
•
If the test fails, we may have to run a separate test on another host
Automating Testing with Knowledge and Reasoning
8
Motivating Scenario
James R. Edmondson
Motivating Scenario Requirements
•
•
•
•
•
•
Ability to sequence tests, preferably without a need for a centralized
server (single point of failure)
Ability to sequence different types of programs (e.g. Python results
parsers, C++ services, and Java smart phone applications)
Ability to disseminate knowledge of test failure and success
Configurable knowledge reasoning service to make sense of the incoming
information
Portable to Windows, Linux and other operating systems
We need fast dissemination and evaluation of real-time results
Automating Testing with Knowledge and Reasoning
9
Related Work
James R. Edmondson
Centralized solutions
Pros
•
•
Central configuration and management
Good for tasks that run once and do not
interact with other distributed tasks
Cons
• Central point of failure
• Bottleneck on test-related messaging
• Most use synchronous messaging which
can cause timing issues when millisecond
precision is required (e.g., repeating race
conditions)
• Usually very host specific and frail to
faults in any of the testing processes
Automating Testing with Knowledge and Reasoning
10
Related Work
James R. Edmondson
Decentralized solutions
Pros
•
•
•
No central point of failure
No central bottleneck in messaging
May use Network File System lockfiles for
host agnostic messaging system
• Great for clouds or LANs
Cons
• NFS lockfiles can have delay of tens or
hundreds of seconds
• NFS file server is usually centralized
• Those that don’t use NFS use hostspecific messaging that is prone to
process/host failure
Automating Testing with Knowledge and Reasoning
11
Solution Approach
James R. Edmondson
Solution Architectural Overview
EXTERNAL DEPENDENCIES
C++ for speed and portability
ACE for portable threads and OS calls
DDS for transport of knowledge
Android SDK for Android smartphone dev
Monkeyrunner for sending events to phones
DEVELOPED TECHNOLOGIES
MAML for instrumentation of phones
KaRL for knowledge and reasoning
KATS for automated testing
MAML
KATS
KaRL
ACE
C++
Automating Testing with Knowledge and Reasoning
DDS
MonkeyRunner
Android SDK
Python
12
Java
Solution Approach
James R. Edmondson
Features of KATS
• Real-time Scheduling via ACE real-time class
• Minimizes latency and jitter by elevating process priority
• Flexible process configuration
•
•
•
•
Executable names, environment variables, working directories
Redirecting standard in, out, and error to user-specified files
Kill times and signals for aborting or signaling processes
Sequencing based on preconditions, postconditions, and
barriers
• Host agnostic and fault tolerant
• Backups of testing entities can be deployed
• Test failure can be detected at run-time and disseminated
• Support for Android smartphones
Automating Testing with Knowledge and Reasoning
13
Solution Approach
James R. Edmondson
Entity Lifecycle for Deployed Processes
1.
Barrier
Application
Launch
2.
Precondition
Postlaunch
6.
3.
Temporal delay
Postcondition
7.
4.
Postdelay
Exit
8.
5.
Automating Testing with Knowledge and Reasoning
Each testing entity has eight life cycle
phases which flow linearly until
completion. Each phase is disabled
by default.
The barrier condition is met when
all potentially distributed entities
participating in a test have
reached the first phase of their
life cycle.
14
Solution Approach
James R. Edmondson
Entity Lifecycle for Deployed Processes
1.
Barrier
Application
Launch
2.
Precondition
Postlaunch
6.
3.
Temporal delay
Postcondition
7.
4.
Postdelay
Exit
8.
5.
Automating Testing with Knowledge and Reasoning
Each testing entity has eight life cycle
phases which flow linearly until
completion. Each phase is disabled
by default.
The precondition is a knowledge
requirement that must be true
before proceeding (e.g.
ServicesStarted for Phone App 1)
15
Solution Approach
James R. Edmondson
Entity Lifecycle for Deployed Processes
1.
Barrier
Application
Launch
2.
Precondition
Postlaunch
6.
3.
Temporal delay
Postcondition
7.
4.
Postdelay
Exit
8.
5.
Automating Testing with Knowledge and Reasoning
Each testing entity has eight life cycle
phases which flow linearly until
completion. Each phase is disabled
by default.
The temporal delay is a platformneutral sleep of at least the
specified number of seconds
16
Solution Approach
James R. Edmondson
Entity Lifecycle for Deployed Processes
1.
Barrier
Application
Launch
2.
Precondition
Postlaunch
6.
3.
Temporal delay
Postcondition
7.
4.
Postdelay
Exit
8.
5.
Each testing entity has eight life cycle
phases which flow linearly until
completion. Each phase is disabled
by default.
Automating Testing with Knowledge and Reasoning
The postdelay is a knowledge
expression to evaluate after the
temporal delay has finished.
17
Solution Approach
James R. Edmondson
Entity Lifecycle for Deployed Processes
1.
Barrier
Application
Launch
2.
Precondition
Postlaunch
6.
3.
Temporal delay
Postcondition
7.
4.
Postdelay
Exit
8.
5.
Automating Testing with Knowledge and Reasoning
Each testing entity has eight life cycle
phases which flow linearly until
completion. Each phase is disabled
by default.
The application launch creates a
heavy weight process to run the
user-specific executable
18
Solution Approach
James R. Edmondson
Entity Lifecycle for Deployed Processes
1.
Barrier
Application
Launch
2.
Precondition
Postlaunch
6.
3.
Temporal delay
Postcondition
7.
4.
Postdelay
Exit
8.
5.
Automating Testing with Knowledge and Reasoning
Each testing entity has eight life cycle
phases which flow linearly until
completion. Each phase is disabled
by default.
The postlaunch phase begins
with the evaluation of userprovided knowledge expressions
and extends until the application
exits or is killed
19
Solution Approach
James R. Edmondson
Entity Lifecycle for Deployed Processes
1.
Barrier
Application
Launch
2.
Precondition
Postlaunch
6.
3.
Temporal delay
Postcondition
7.
4.
Postdelay
Exit
8.
5.
Automating Testing with Knowledge and Reasoning
Each testing entity has eight life cycle
phases which flow linearly until
completion. Each phase is disabled
by default.
The postcondition is evaluated
after the application has exited. A
return value is made available to
the user-specified knowledge
expression
20
Solution Approach
James R. Edmondson
Entity Lifecycle for Deployed Processes
1.
Barrier
Application
Launch
2.
Precondition
Postlaunch
6.
3.
Temporal delay
Postcondition
7.
4.
Postdelay
Exit
8.
5.
Automating Testing with Knowledge and Reasoning
Each testing entity has eight life cycle
phases which flow linearly until
completion. Each phase is disabled
by default.
During the exit phase, the testing
entity cleans up and ends its life
cycle
21
Solution Approach
James R. Edmondson
Modeling Deployed Processes
A GME DSML paradigm is provided for modeling all testing
entities:
• Visual interface for configuring entities and their life cycles
• The modeled entity attributes are generated into an XML file
• Recursive includes are allowed to chain entity groups and
provide for intricate testing sequences
• Provides specialized KATS processes like the KATS Observer,
which allows for knowledge monitoring of active tests
Automating Testing with Knowledge and Reasoning
22
Case studies
James R. Edmondson
Case Study #1
• One gateway service
– Requires 15 seconds to load
databases before servicing
requests
• Five Android smartphones
– Wait for 15 seconds
– Race condition has been
detected when smartphones
connect sequentially within
microseconds
– Need to sequence the five
phones to reproduce the race
condition
Name
Pre
Postlaunch Delay
Gateway
Gateway=1
Phone1
Gateway
Phone1=1
Phone2
Phone1
Phone2=1
Phone3
Phone2
Phone3=1
Phone4
Phone3
Phone4=1
Phone5
Phone4
Automating Testing with Knowledge and Reasoning
23
15
Case Studies
James R. Edmondson
Case Study #2
•
Two gateway services
– One main
– Backup doesn’t launch unless
main dies or phones can’t
connect to main
– Kill 180 seconds after launch
•
Five phones
– Three connect to the main
– Two connect if the main
phones return error codes or if
the backup is launched
– Kill 180 seconds after launch
Name
Precond.
Main
Backup
Postcondition
Kill
.kats.return=> backup=1
180
backup
180
Phone1
.kats.return=> backup=1
180
Phone2
.kats.return=> backup=1
180
Phone3
.kats.return=> backup=1
180
Phone4
backup
180
Phone5
backup
180
Automating Testing with Knowledge and Reasoning
24
Experiment Results
James R. Edmondson
Dissemination Latency for Conditions
•
•
•
Result is based on roundtrip
latency so clock synchronization is
not an issue
Bottleneck is transport
Default transport with KATS is
DDS
Automating Testing with Knowledge and Reasoning
25
Experimental Results
James R. Edmondson
Barrier Latency
•
•
•
•
Latency is the time between
entering and exiting the barrier
Test is repeated 10 times and
statistics are reported to the right
The latency is almost completely
dependent on operating system
jitter in launching the processes
simultaneously
Machines only had 2 processors
Automating Testing with Knowledge and Reasoning
26
Conclusion
James R. Edmondson
Future Work
• Redeployment
–
–
–
–
–
Deployment may become inadequate
Application must be redeployed
Side effects (like databases) can complicate the migration
Constraint satisfaction problem (NP complete)
We’ve created an approximation technique called CID
•
•
•
•
O(N2) (100 ms for 4,000 deployed entities)
Not perfect but does better than known local search techniques
May be able to use this for redeployment in continuous, real-time systems
Must take side effects into account
Automating Testing with Knowledge and Reasoning
27
Conclusion
James R. Edmondson
Concluding Remarks
•
MADARA KATS provides an extensible automated testing environment
– Portable to Windows, Linux, Mac, etc.
– Allows specification of executable, command line, kill times and
signals, elevation of processes to highest scheduling priority, and
much more
– Provides a fine-grained process lifecycle for strict test execution
– Supports XML input and batch processing
– Can support testing or deployment of any distributed application
•
MADARA KATS is open source
– BSD license and download at http://madara.googlecode.com
– Questions or comments? jedmondson@gmail.com
Automating Testing with Knowledge and Reasoning
28