Tips for Expanding Your
Testing Toolbox
ALAN PAGE
Ground Rules, etc.
Agenda
Security
Performance
Scripting
Code Review
Tools
Etc.
Generalizing Specialists
1.A generalizing specialist is someone who has one or more
technical specialties (Has at least a general knowledge of
software development).
2.Has at least a general knowledge of the business domain in
which they work.
3.Actively seeks to gain new skills in both their existing
specialties as well as in other areas, including both technical
and domain areas.
http://www.agilemodeling.com/essays/generalizingSpecialists.htm
“A generalizing specialist is more than just a
generalist. A generalist is a jack-of-alltrades but a master of none, whereas a
generalizing specialist is a jack-of-all-trades
and master of a few.”
- Scott Ambler
Security
Threat
Description
Asset
Impact
Master secret server is a
single point of failure
If a malicious user
compromises the master
secret server, the SSO
BizTalk Server and SSO
computer is unable to
environment
encrypt credentials (it is
able to continue decrypting
credentials).
Denial of service
A malicious user can spoof
a client or server
If a client or server runs
Windows without NTLM
authentication, a malicious
user can spoof the client or
server.
BizTalk Server and SSO
environment
Spoofing identity
Tampering with data
Repudiation
Information disclosure
Denial of service
Elevation of privileges
A malicious user can
tamper with the data as it
travels from one server to
another
The communication
between servers is in clear
text, and a malicious user
can potentially read the
data as it travels.
Data
Tampering with data
Information disclosure
INSERT INTO Students VALUES ( '$Name' )
INSERT INTO Students VALUES ( 'Robert' );
DROP TABLE STUDENTS; --')
Add a new record to the Students table with a Name value of ‘Robert’
- and –
Delete the Students table
http://stackoverflow.com/questions/332365/how-does-the-sql-injection-from-the-bobby-tables-xkcd-comic-work
More Examples
1=1
UserID: 101 or 1 = 1
SELECT * FROM Users WHERE UserID = $UserID
SELECT * FROM Users WHERE UserID = 101 or 1 = 1
Batch
UserID: 101; DROP TABLE Grades
txtSQL = "SELECT * FROM Users WHERE UserId = " + txtUserId;
SELECT * FROM Users WHERE UserId = 101; DROP TABLE Grades
XSS (Cross Site Scripting)
Goal – get arbitrary JavaScript to run
<script>alert(“haXX0r!”)</script>
Most modern browsers guard against (some) XSS
Statistics and Math in
Software Testing
Average
Data set A
(5×1 + 10×2 + 20×3 + 30×4 + 20×5 + 10×6 + 5×7) ÷ 100 = 4
Data set B
(80×1 + 20×16) ÷ 100 = 4
Data set C
(10×1 + 11×2 + 13×3 + 11×4 + 11×5 + 11×6 + 12×7 + 10×8) ÷ 100 = 4
Medians & Normal Values
Median
the middle value in a data set when sequenced from lowest to highest
Normal Value
single value that occurs most often in a data set
Data Set
Median
Normal Value
A
4
4
B
1
1
C
4
3
Standard Deviations
Formula
S = Sqr(∑((Xi – average)^2) ÷ (n – 1))
Common rule: “Data with a standard deviation greater than half of its mean should be treated
as suspect.”
Data Set
Standard Deviation
Mean
A
1.5
4
B
6.0
4
C
2.6
4
Normal Distributions
Visualizing Data
Once we understand our data, visuals
can help convey the tendency of the
data quickly
Histograms are used to understand the
frequencies of the observations
◦ Unimodal Distribution
◦ Bimodal Distribution
Scatterplots are used to understand
correlation
Why we start with understanding the data
CPU Usage Over Time
700%
600%
CPU Usage vs Baseline
500%
400%
Mean
300%
200%
100%
0%
12/31
1/1
1/2
1/3
1/4
1/5
1/6
2014
1/7
1/8
1/9
1/10
1/11
Try Looking At It Another Way
CPU Usage Over Time
700%
600%
CPU Usage vs Baseline
500%
400%
Mean
300%
Median
200%
100%
0%
12/31
1/1
1/2
1/3
1/4
1/5
1/6
2014
1/7
1/8
1/9
1/10
1/11
Try a Histogram
Bucketized CPU Usage Observations
3.5
3
Frequency
2.5
2
1.5
1
0.5
0
0%
100%
200%
300%
400%
CPU Usage vs Baseline
500%
600%
More
Look Again
CPU Usage Over Time
700%
600%
CPU Usage vs Baseline
500%
400%
Observations
Mean
300%
Median
200%
100%
0%
12/31
1/1
1/2
1/3
1/4
1/5
1/6
2014
1/7
1/8
1/9
1/10
1/11
Scripting / Coding
Discussion points
Not just for test automation
Automate repetitive / boring stuff
Choosing a scripting language
Code Reviews
Code Review – Cheat Sheet
Ask about ifs without elses
Magic numbers
Are there enough tests – do they do something useful?
What happens if…?
Ask anything (especially, “How would you test this?”
Confusing is confusing
Operations in loop
Peering Into The White Box: A Testers Approach to Code Reviews
#include <string.h>
char *strcpy (char *dest, char *src);
void bcopy (void *src, void *dst, int n);
HWND CreateWindow(LPCTSTR lpClassName, LPCTSTR lpWindowName, DWORD
dwStyle, int x, int y, int nWidth, int nHeight, HWND hWndParent, HMENU
hMenu, HINSTANCE hInstance, LPVOID lpParam);
int CheckValueAndUpdateIfNeeded(structure s)
Other Tools
Static Analysis tools
◦ Examine code and find common coding errors
◦ Examples are Code Analysis in Visual Studio, and SonarQube / FindBugs for Java
Dynamic Analysis tools
◦ Examine running applications looking for subtle programming errors
◦ Examples are FXCop (managed code), Application Verifier (for windows), and GlassBox for
Java, Valgrind for Linux, etc.
(Relatively) simple to run (depending on tool), but are great at finding bugs
http://angryweasel.com
http://angryweasel.com/ABTesting
alan.page@microsoft.com
Twitter: @alanpage
Cucumber (BDD)*
Jira
Selenium / Webdriver*
Xmind*
Silk
Hexawise*
TestNG
Browserstack
Dev tools in Browser
Jing
OpenBroadcast
Visual Studio
Diff tools*
HP Quality Center
TFS
Fiddler*
Postman