Noelle A. Stimely
Senior Performance Test Engineer
University of California, San Francisco
noelle.stimely@ucsf.edu
Who am I?






Senior Oracle Database Administrator for over 13 years
Senior Performance Test Engineer
Past Oracle Instructor
Past Lead Database Administrator
Oracle Certified Professional (OCP) – 8, 8i, 9i,10g
Hewlett Packard AIS – LoadRunner v11Certified
2
University of California, San Francisco

University of California, San Francisco is
a leading university dedicated to
defining health worldwide through
advanced biomedical research,
graduate level education in the life
sciences and health professions, and
excellence in patient care
◦
◦
◦
◦
3000+ Students
20,000+ faculty, staff and post docs
Annual budget about $2.4 billion
San Francisco's second largest
employer
3
This is not going to be a presentation on database tuning,
but on load testing.
4
Stressed out due to performance issues?
5










What is load testing?
Types of load tests
Purpose of load testing
Developing the load test
Load test design
Load testing tools
The test itself
Analyze and report
Conclusions
DEMO
6
What is Load Testing?
7
Types of Load Tests

Transaction response testing
◦ Order entry must complete within 8 seconds
◦ AP query must return results within 5 seconds
◦ PDF attachment must upload within 5 seconds

End user experience testing
◦ Run full-scale web load test while a subset of users logs into system to conduct
normal work activity
◦ Have a subset of end users log into the system in the middle of load test to gauge
performance

Stress test
◦ Multiple users logging into the system at one time (100 users log in at one time)
◦ Users logging into the system very rapidly (e.g. 1 user every second)
◦ Extended concurrency times (25 users remain logged into system running heavy
transactions for an extended period)
8
Why load test?






After a software/hardware upgrade?
Does the application perform acceptably with a minimum amount of
users on the system?
Does the application respond quickly with additional concurrent
users?
Are there hardware bottlenecks in the system?
Can the application handle a growing number of users/data?
What is the maximum amount of load the system can handle?
9
Developing the Load Test
10
Load Test Team







Application owner(s)
Database administrator(s)
Web server administrator(s)
Network administrator(s)
System administrator(s)
Developer(s)
Project manager(s)
11
Load Test Criteria Identification
12
Load Test Criteria Identification (cont’d)

Test environment
◦
◦
◦
◦
Close to production as possible (ideal)
How to adjust for inconsistencies between test and production
Building environment
Testing the test environment
13
Load Test Criteria Identification (cont’d)

Performance acceptance criteria
◦
◦
◦
◦
◦
◦
◦
◦
Service Level Agreements (SLAs)
End user expectations
Business requirements
Workload – key scenarios representing reality
Performance optimization objectives
Scalability
Range of accepted workload conditions
Resource utilization objectives
14
Load Test Criteria Identification (cont’d)

Application Scenarios
◦ What aspect of application going to test?

Breakdown into separate scenarios or tests?

Actions to be performed?
◦ Example – Book a flight:

Log into application

Search airline flights

Book flight

Pay for flight with credit card

Logout of application
◦ Scenario mix
◦ Screenshots of scenario
15
Load Test Criteria Identification (cont’d)

Application Scenarios (cont’d)
◦ Screenshot example:
16
Load Test Profile

Performance Characteristics
◦ How many users have been concurrently logged into the system at the busiest time?
◦ How many concurrent users will be used for the test?
◦ How will users be divided among the script tasks to be tested, i.e. test 4 actions
during test and divide users among scripts – 100 for script 1, 200 for script 2, 50 for
script 3 and 150 for script 4.
◦ Number of users: the total number of concurrent and simultaneous users who access
the application in a given time frame
◦ Rate of requests: the requests received from the concurrent load of users per unit
time.
◦ Patterns of requests: A given load of concurrent users may be performing different
tasks using the application. Patterns of requests identify the average load of users,
and the rate of requests for a given functionality of an application.
◦ Will user pause before re-logging in again to perform action, i.e. wait 30 seconds
before logging in again?
◦ How long will users spend on each web page of application during test (think time)
◦ How long will the test run?
17
Application Usage Patterns

Scenario
◦
◦
◦
◦
Search and book flight.
Search for flights without booking.
Change an existing reservation.
Ticket agent running a report.
Script
Search and book flight
Search for flights without
booking
Change an existing reservation
Ticket agent running a report
Total
% of users
20
60
Total
200
600
10
10
100
100
100
1000
18
Load Test Profile Considerations

User activity
◦

Ramp-up time
◦

The time it takes for a user to respond to an application page has a significant impact on the number of
users the application can support without performance issues.
Run Length
◦

Load test ramp down implies gradually stopping the threads during a load run in order to detect memory
leaks and check system recovery after the application has reached a threshold.
Think Times
◦

A major component of stress on a site/application is how many and how quickly users log into the system.
Ramp-down time
◦

Some transactions occur more often than others and therefore should be a larger part of the test data and
scenarios.
The length of the actual load test can vary based on many factors. You should aim to run the test long
enough to get a representative example of true system performance.
Workload/users
◦
Calculating the number of users for the load test is critical in order to have a valid test which accurately
forecasts performance.
19
Load Profile
Scenario 1 –
20 Users
Script
Script0
1
Action
Search &
Book
Flight
Search
for
flights
Script0 without
2
booking
Change
existing
Script0 reservati
3
on
Load Profile
#
Iteration
s
Think
Time sec
Random
of
recorded
think
time: 50%
- 150%.
Limit 10
1 seconds.
Random
of
recorded
think
time: 50%
- 150%.
Limit 10
1 seconds.
Random
of
recorded
think
time: 50%
- 150%.
Limit 10
1 seconds.
# Users
Ramp Up
# Users
Ramp
Down
0
10 Seconds
1 User
every 10
seconds
1 User
every 10
seconds
30
0
7 Seconds
1 User
every 10
seconds
1 User
every 10
seconds
30
0
3 Seconds
1 User
every 10
seconds
1 User
every 10
seconds
30
User
Pacing
Duration
(mins)
20
Load Test Design




Application accounts are created for the test with appropriate
privileges.
Test data feeds are properly implemented for testing. Test data
feeds are data which is inputted into the system during the load test
to create variances in activity thus mimicking real life.
Validation of scenario transaction on the webserver, database,
application server and hosting server.
Special data used in application which needs to be incremented for
proper execution such as product ID.
21
Test IDs






Create test id(s)
Application owner role
Appropriate privileges
ID data masking
Unique ids
Parameterization of values
Login ID
Test040
Password
pass01
Test041
pass01
Test042
Test043
Test044
Test045
Test046
Test047
Test048
Test049
Test050
Test051
Test052
Test053
Test054
Test055
Test056
Test057
Test058
Test059
Test060
Test061
Test062
Test063
Test064
Test065
Test066
Test067
Test068
Test069
Test070
Test071
Test072
Test073
pass01
pass01
pass01
pass01
pass01
pass01
pass01
pass01
pass01
pass01
pass01
pass01
pass01
pass01
pass01
pass01
pass01
pass01
pass01
pass01
pass01
pass01
pass01
pass01
pass01
pass01
pass01
pass01
pass01
pass01
pass01
pass01
22
Test Data Feeds








What is test data?
Where is test data used?
How to use test data
Creation of test data
Parameterization of test data
Unique/non-unique data
Oracle Flashback
Data restore for non-Oracle systems
23
Test Data Feed example:
Test Scenario
ID #
login id
name
pwd
TS Group
BWApprover
BWApprover
1
025985565 MOOS,BRIAN
123456
848066
BWApprover
2
020012209 ABBY,JOHN S
123456
747002
BWApprover
3
021280441 ARRY,MONICA
123456
875001
BWApprover
4
002203122 HEBERT,JENNIFER SCHOE
123456
865009
BWApprover
5
024221582 MANDLE,JANE
123456
787091
BWApprover
6
022502256 FLANN,FRANNIE
123456
861090
BWApprover
7
026555102 SEIE,MARY A
123456
729090
BWApprover
8
024545188 KLING,GERRY A
123456
750004
BWApprover
9
022704886 MASA,ALISHA FUMI
123456
863008
BWApprover
10 028351336 THOR,JILEL
123456
874030
BWApprover
11 028389732 LONG,AMIE
123456
774002
BWApprover
12 029211083 KNOL,KATHY M
123456
763005
BWApprover
13 021754882 FRANCIS,RONNIE M
123456
763001
BWApprover
14 023307119 JUANNY,DIANE S
123456
763420
BWApprover
15 028862555 RAGOE,KATHY A
123456
763006
BWApprover
16 028577427 VEGAS,SUSY
123456
763002
Pay Period
08/10/08 08/23/08
08/10/08 08/23/08
08/10/08 08/23/08
08/10/08 08/23/08
08/10/08 08/23/08
08/10/08 08/23/08
08/10/08 08/23/08
08/10/08 08/23/08
08/10/08 08/23/08
08/10/08 08/23/08
08/10/08 08/23/08
08/10/08 08/23/08
08/10/08 08/23/08
08/10/08 08/23/08
08/10/08 08/23/08
08/10/08 08/23/08
Employee
Date
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
Notes
24
Load Testing Tools – Freeware vs. Commercial

Freeware
◦
◦
◦
◦

Jmeter
HammerOra
Swingbench
OpenSTA
Commercial
◦ HP Loadrunner
◦ Oracle Application Testing Suite
◦ SilkPerformer
25
Oracle Real Application Testing
vs
Load Testing





What is Oracle Real Application Testing (RAT)?
Why RAT should not be used by itself for formal load testing.
How is RAT used?
Oracle Real Application Testing Suite
How to use Real Application Testing Suite with RAT to create load
tests
26
Monitoring the Load Test

Identify metrics to be used for the load test
◦ OS metrics which could include CPU, memory, disk I/O and network I/O statistics.
◦ Database metrics to watch for are waits, top SQL, Buffer gets, etc.
◦ Network metrics would include the overall health of your routers, switches and
gateways.
◦ Webserver metrics would include hits per second, number of requests, etc.

Identify team members who will monitor during the test
◦
◦
◦
◦
What areas will be monitored?
What tools will be used?
How will the tools be used to monitor (invasive/non-invasive?)
How will the results be reported at the conclusion of the test?
27
Analyze and Report Test Results

Pinpoint bottlenecks found during load test
◦
Examples:









Checkout should not take longer than 4 seconds under 15 hits per second workload
CPU utilization for the test should not exceed 80% during any transaction
Transactions in average response time graph breach the performance
acceptance criteria.
Response time that is worse than what the SLA for the customer allows.
Correlate transactions with other graphs and measurements to find
bottlenecks.
Involve developers, system and network administrators in investigation.
Overlay response time graph with monitoring graphs and look for
performance trends.
Check standard deviation of transactions if transactions behave
inconsistently identify the cause.
Transactions availability: (count of pass transactions/count of total
transactions) x 100. Usually applications have around 99.9% availability.
Lower values don’t necessarily indicate an issue, but should be verified
why lower.
28
Load Test Report Graphs

LoadRunner graph showing a performance irregularity occurred
between 14 and 18 minutes into the test
29
Hits per Second Graph – LoadRunner

Same load test as previous slide – Note drop off in hits at 15 minute
point in test
30
Reporting Audience

Management/Project Managers/Team Leads?
◦ Technical aptitude
◦ Length of report
◦ Type of report

Suggested Graphs to include in report
◦ Running Users – Shows the impact of ramp-up and ramp-down as well as when
errors begin to occur during the load test.
◦ Hits per Second – Shows the amount of load generated by virtual users in terms of
hits to web servers.
◦ Throughput – Shows the amount of load virtual users create on network resources –
raw amount of bytes client receives each second as a result of hits on the web server.
◦ Average Transaction Response Time – Shows the actual load generated by each
transaction throughout the entire load test.
31
OpenSTA Graph Reporting


Graphs can be customized for presentation purposes
Data from graphs can be exported to Excel for further analysis
32
Load Test Summary

Load test summary report
◦ Top 5 Slowest Pages

Slowest pages encountered during load test
◦ Top 5 Slowest Transactions

Do not meet the performance criteria defined at test inception
◦ Top 5 Slowest SQL Statements

Slowest SQL which is typically 80% of performance problems
33
Find Bottlenecks, Tune System – Retest?
34
CONCLUSIONS
35
DEMONSTRATION
36
www.collaborate12.org
www.collaborate12.ioug.org