Multi-threading Testcases with SilkTest
Contents
Introduction.................................................................................................................................................. 2
The ‘parallel’ Statement............................................................................................................................. 2
The ‘multitestcase’ Statement .................................................................................................................. 4
The SetUpMachine() Statement .......................................................................................................... 4
The ‘spawn’ Statement and the ‘setMachine’ Statement ................................................................. 5
The ‘rendezvous’ Statement ................................................................................................................. 6
Alternative Launch Syntax .................................................................................................................... 7
Thread Control ............................................................................................................................................ 7
The ‘share’ and ‘access’ statements ................................................................................................... 7
The ‘critical’ Statement .......................................................................................................................... 8
The ‘semaphore’ Datatype .................................................................................................................... 8
Introduction
SilkTest has long had support for multi-threaded testcases but the caveat is that it only works for
remote parallel testing. In other words, you cannot run multiple threads on a local testcase, nor
can you run multiple threads on a single remote machine. This only works for two or more
machines being run remotely.
There are two ways to set up a multi-threaded test:


Using the ‘parallel’ statement
Using the ‘multitestcase’ statement
You should also note that multithreaded remote testing does not imply that all machines should
be running the same application under test. Each machine can be running an entirely different
test and the results will be returned to the launch (host) PC.
There are some differences between the two calling methods which you must be aware of. The
following five control methods are unique to the multitestcase method:





SetUpMachine()
SetMultiAppstates()
SetMachine()
spawn
rendezvous
In addition, there are a set of control methods that are common to both multitestcase and
parallel calls:






access
acquire
critical
release
semaphore
share
We will discuss each of these in turn, but first we will look at the two methods of starting a
multithreaded testcase.
The ‘parallel’ Statement
If you choose to use the parallel statement you must be aware of two additional caveats:


The test MUST be launched from a main() function and
it does not invoke the Recovery System.
The second is not quite so much of a problem as it first appears because if you are calling
testcases inside your main() function you can create your own Recovery System using the
TestcaseEnter() or TestcaseExit() methods to ensure that the application under test is in a state
fit to start testing. See the Help files topic ‘main function, using in a script’ for how to call a
testcase from a main() function.
The advantage of the ‘parallel’ statement over using ‘multitestcase’ is that it implicitly waits for
all threads to complete at the end of the multithreading block before continuing on with the test.
Multitestcase requires that you explicitly tell SilkTest where the end of the multithreading block
is (see below). The following illustrates the outline of a parallel testcase using a remote server to
supply information to the remote clients. The requirement to only multithread on remote
machines does not stop you from calling the local machine as part of the test. This will work
correctly as long as the calls to the local (host) PC are outside the ‘parallel’ code blocks.
[-] main ()
[ ] HMACHINE client1_handle
[ ] HMACHINE client2_handle
[ ] HMACHINE server_handle
[ ]
[ ] server_handle = Connect (“server”)
[ ] client1_handle = Connect (“client1”)
[ ] client2_handle = Connect (“client2”)
[ ]
[ ] SetupServerState (“server”)
[ ]
[ ] // Start the multithreaded calls to the remote PCs
[-] parallel
[ ] RunMyTest (“client1”)
[ ] RunMyOtherTest (“client2”)
[ ] // The testcase will halt here until both client1 and client2 are done
[ ]
[ ] // The testcase is done so tidy-up
[ ] TidyUpServer (“server”)
[ ]
[ ] //Clean up behind yourself
[ ] Disconnect (server_handle)
[ ] Disconnect (client1_handle)
[ ] Disconnect (client2_handle)
[ ]
[ ] //Testing ends here
[ ]
[ ] //---------------------------[ ] //Actual test code follows
[ ]
[ ] //Setup the server
[-] SetupServerState (STRING sMachineName)
[ ] SetMachine(sMachineName)
[ ] <do some setup actions here>
[ ] <e.g. populate some variables required by the remote PCs>
[ ]
[ ]
[ ] //Call the first client
[-] RunMyTest (STRING sMachineName)
[
[
[
[ ]
[-]
[
[
[
[ ]
[-]
[
[
[
] SetMachine(sMachineName)
] <do some testing actions here on the first application>
]
//Call the second client
RunMyOtherTest (STRING sMachineName)
] SetMachine(sMachineName)
] <do some testing actions here on the second application>
]
// Return the results to the server and clean up
TidyUpServer (STRING sMachineName)
] SetMachine(sMachineName)
] <clean up the server>
]
The other big disadvantage of using the ‘parallel’ statement is that you cannot call an appstate;
again, this can be alleviated by creating your own appstate in the TestcaseEnter() method as
long as you are calling testcases inside the main() method. You cannot do this in the example
above as it runs directly from the main() method without any internal testcases.
The ‘multitestcase’ Statement
The multitestcase method has the advantage that you have access to the Recovery System and
that you can call an appstate during the setting up procedure. The testcase outline is not the
same as the parallel statement above although it does have similarities. Rather than provide the
entire outline of a mulitestcase, we will build it up section by section. Each of the following
sections will outline the methods that are unique to multitestcase.
The SetUpMachine() Statement
This is the only unique statement that has more than one parameter:[ ] SetUpMachine(sMachine,wMainWin(optional),sAppState(optional))
Before you can run any code on the remote PCs you must enable the remote PCs using the
SetUpMachine() method. The method connects to the remote machine and prepares it through
the specified appsatate before executing any test code. SetUpMachine() implicitly issues the
Connect() statement which you need to explicitly state if using the parallel method. Although
wMainWin and sAppState are optional, if you do not declare a wMainWin then SilkTest cannot
invoke the Recovery System as the Recovery System needs to know the application’s main
window (see the Recovery System in any frame file). By careful choice of variables you can
effectively black-box the calls to SetUpMachine() so that a simple change to a set of global
variables will call a whole new set of remote machines and/or different appstates.
[
[
[
[
[
[
]
]
]
]
]
]
//List
string
string
string
the PCs that will be running the tests
sRemotePC1=”Pippin”
sRemotePC2=”Frodo”
sRemotePC3=”Bilbo”
//List the appstates we want to start with
[ ]
[ ]
[ ]
[ ]
[ ]
[ ]
[ ]
[ ]
[ ]
[-]
[
[
[
[
string sAppState1=”TheParty”
string sAppState2=”GoToMordor”
string sAppState3=”DestroyTheRing”
// List the main windows for the recovery system
string sMainWin1=”GoldenPerch”
string sMainWin2=”PrancingPony”
string sMainWin3=”WellingHall”
multitestcase Adventure (string sRemotePC1,string sRemotePC2,string sRemotePC3)
] SetUpMachine(sRemotePC1,sMainWin2,sAppState3)
] SetUpMachine(sRemotePC2,sMainWin1,sAppState2)
] SetUpMachine(sRemotePC3,sMainWin3,sAppState1)
] SetMultiAppStates()
You must terminate the SetUpMachine() calls with a call to SetMultiAppStates() to ensure that
the application(s) are driven to the correct state before starting the test. Once the
SetUpMachine functions are completed, you can go on to start the threads.
The ‘spawn’ Statement and the ‘setMachine’ Statement
The spawn statement notifies SilkTest that this is the start of the multithreaded code. Once a
spawn method has been called we need to make a call to the individual PC via the setMachine()
statement to notify it that test code follows. If you have done any serial remote testing then the
setMachine() statement may already be familiar to you.
Each machine called in the SetUpMachine() block must have a spawn statement and each
spawn statement must have a setMachine() call.
[-] multitestcase Adventure (string sRemotePC1,string sRemotePC2,string sRemotePC3)
[ ] SetUpMachine(sRemotePC1,sMainWin2,sAppState3)
[ ] SetUpMachine(sRemotePC2,sMainWin1,sAppState2)
[ ] SetUpMachine(sRemotePC3,sMainWin3,sAppState1)
[ ] SetMultiAppStates()
[ ]
[-] spawn
[ ] setMachine(sRemotePC1)
[ ] <run the test code>
[-] spawn
[ ] setMachine(sRemotePC2)
[ ] <run the test code>
[-] spawn
[ ] setMachine(sRemotePC3)
[ ] <run the test code>
The obvious difference here is that we are running the test code inside the spawn statement.
However, there is nothing to stop you from using the same test code structure as we have
shown in the parallel statement:[-]
[
[
[
multitestcase Adventure (string sRemotePC1,string sRemotePC2,string sRemotePC3)
] SetUpMachine(sRemotePC1,sMainWin2,sAppState3)
] SetUpMachine(sRemotePC2,sMainWin1,sAppState2)
] SetUpMachine(sRemotePC3,sMainWin3,sAppState1)
[ ]
[ ]
[-]
[
[-]
[
[-]
[
SetMultiAppStates()
spawn
] RunMyTest1 (sRemotePC1)
spawn
] RunMyTest2 (sRemotePC2)
spawn
] RunMyTest3 (sRemotePC3)
//------//
[ ]
[-]
[
[
[
[-]
[
[
[
[-]
[
[
RunMyTest1 (STRING sMachineName)
] SetMachine(sMachineName)
] <run the test code>
]
RunMyTest2 (STRING sMachineName)
] SetMachine(sMachineName)
] <run the test code>
]
RunMyTest3 (STRING sMachineName)
] SetMachine(sMachineName)
] <run the test code>
The ‘rendezvous’ Statement
The rendezvous statement notifies SilkTest that this is the end of the multithreading block. As
threads will run at different speeds on different machines depending on hardware and load, the
rendezvous statement stalls each thread as it returns until the last thread has completed its
testing. At this point SilkTest releases the testcase to continue running.
[-] multitestcase Adventure (string sRemotePC1,string sRemotePC2,string sRemotePC3)
[ ] SetUpMachine(sRemotePC1,sMainWin2,sAppState3)
[ ] SetUpMachine(sRemotePC2,sMainWin1,sAppState2)
[ ] SetUpMachine(sRemotePC3,sMainWin3,sAppState1)
[ ] SetMultiAppStates()
[ ]
[-] spawn
[ ] setMachine(sRemotePC1)
[ ] <run the test code>
[-] spawn
[ ] setMachine(sRemotePC2)
[ ] <run the test code>
[-] spawn
[ ] setMachine(sRemotePC3)
[ ] <run the test code>
[ ]
[ ] rendezvous
[ ]
[ ] <more testing here>
If the testcase terminates at the end of the multithreading block then there is no real need to use
a rendezvous statement. However, it is recommended to do so if only for the sake of neatness
in the script.
Alternative Launch Syntax
There is an alternative syntax that can be used when starting a multitestcase but it is better
suited to running identical remote tests. It draws the machine name from a list of string and by
way of a loop, launches each remote machine until it reaches the end of the list.
[-] multitestcase runMulti (list of string lsmachines)
[ ] string sMachine
[ ]
[-] for each sMachine in lsMachines
[-] spawn
[ ] SetUpMachine(sMachine,”wAppMainWin”,”sAppstate”)
[ ] rendezvous
[ ]
[ ] SetupMultiAppstates()
[ ]
[-] for each sMachine in lsMachines
[-] spawn
[ ] [sMachine]PerformSomeActivity()
[ ] rendezvous
[ ] <etc..>
Note that the application’s main window name and the starting appstate are hard-coded into the
SetUpMachine() call. It would be possible to also draw these from a list of string so that different
appstates and mainwins can be loaded each time, but at this point it would be a lot easier to use
the starting layout defined above.
Thread Control
The following thread control methods apply to both the parallel and multitestcase methods of
starting a multithreaded testcase.
The ‘share’ and ‘access’ statements
All threads have equal access to both local and global variables. Uncontrolled access of this
type is a bad thing because you can get multiple threads all trying to update a given variable at
the same time, leaving it in an undefined state that can wreck the rest of your testcase.
Unfortunately we don’t have a precise method of controlling access to local variables except by
careful coding and paying attention to the scope of the variable. For global variables (those that
can be seen by multiple testcases as well as multiple threads) there are the ‘share’ and ‘access’
control methods. These two work hand-in-hand to prevent unwanted changes to a variable.
The syntax for the share method is as follows:<scope> share <variable type> <variable name>=<initial value>
In this case <scope> can be either public or private. If <scope> is omitted then it defaults to
public. <variable type> can be of a standard SilkTest type or it can be a user-defined type.
The variable must be initialised before you try to use it as if it is not, SilkTest will raise a ‘not
defined’ error message when you try to access it. Once you have declared a variable as shared
you will only be able to change or read it by use of the access statement.
The access statement controls access to a shared variable by blocking all other threads that
require to use the variable until the current controlling thread has finished with it. Once the
access script block is exited, SilkTest releases any blocked threads and the variable is available
for access again by the next thread that requires it.
[ ] private share integer iTestNum=0
[ ]
[ ] // Function to update the testcase count
[-] void IncrementTestNum()
[ ] // Block unwanted threading
[-] access iTestNum
[ ] // Increment the test count
[ ] iTestNum++
[ ]
[ ] // Exit access control and release the threads
[ ] return
The ‘critical’ Statement
The ‘critical’ statement acts in a similar fashion to the ‘access’ method but with one important
difference; as soon as a thread enters a critical code block all other threads are immediately
halted wherever they are in their code execution. The threads are released only when the active
thread exists the critical block. Only one thread can have critical status at any one time.
[ ] // Function to update the testcase count
[-] void IncrementTestNum()
[ ] // Halt all threads while this block executes
[-] critical iTestNum
[ ] // Increment the test count
[ ] iTestNum++
[ ]
[ ] // Exit critical control and release the halted threads
[ ] return
The ‘semaphore’ Datatype
The semaphore is the last method associated with multithreaded testcases.
The semaphore datatype has two children – ‘ acquire’ and ‘release’. Semaphores are used to
control access to a resource or to mutually exclude competing threads. Like the ‘shared’
method, the number of semaphores needs to be initialized before trying to use them or else an
exception will be raised on first access.
You cannot directly check for the number of semaphore resources available or in use; to do this
you must typecast the number of semaphores and calculate as necessary.
[ ] semaphore semControl=10
// Allocate ten semaphore flags
[-] if (semControl==[semaphore]2) // If there are only two semaphores left
[ ] print(“semControl is down to {semControl} unused flags”)
‘Acquire’ is used to allocate a semaphore to a requesting thread. If SilkTest has more threads
than semaphores and threads are requesting more semaphores than are available, SilkTest will
block all threads until a semaphore is released. When semaphores are released, SilkTest
allocates the released semaphores in the order that the threads invoked the ‘acquire’ method.
SilkTest then unblocks the thread that it has just given the newly-released semaphore to. If no
threads are waiting for a semaphore when a ‘release’ statement is executed then the
semaphore is put back in the available pile.
‘Release’ is the keyword to actually release an acquired semaphore..
There is a potentially serious problem with the ‘acquire’ child method. If the script experiences
an exception (as opposed to an expected and trapped error) between the ‘acquire’ and ‘release’
statements the script will deadlock with no chance of recovery. To prevent this from happening
wrap any code that could cause a potential exception in the ‘do’ part of a do-except block and a
‘release’ statement in the ‘except’ part.
Probably the best use of semaphores is to control access to a database where a limited number
of users are allowed online at any one time. In the following example we will assume that the
user limit for accessing the database is three users.
[ ] semaphore semDBAcess=3
// Declare three semaphores as available
[-] // Do code here to spawn thread one
[ ] <do some work here>
[ ]
[ ] acquire(semDBAcess)
// Thread one acquires a flag
[ ] print (“Semaphore requested – {semDBAccess} flags left”)
[ ]
[ ] print(“Beginning database access”)
[-] do
[ ] <access the database>
[ ] <update the database>
[-] except
[ ] // The database has thrown an exception
[ ] release (semDBAccess)
[ ] LogWarning (“Unexpected database exception”)
[ ] print (“Semaphore released – {semDBAccess} flags available”)
[ ]
[ ] <do some more work here>
[ ] release (semDBAccess)
// The semaphore is finished with
[ ] print (“Semaphore released – {semDBAccess} flags available”)