Evaluating Performance
Chapter 1
Section 1.4
Dr. Iyad F. Jafar
Outline
 Introduction
 CPU Execution Time
 The Performance Equation
 Determinants of Performance
 SPEC Benchmark
 Other Performance Metrics
 Examples
2
Introduction
 Given a collection of computers, which one to buy?
 Best Performance ?
 Least Cost ?
 Best Cost/Performance?
 How to define performance?
 Time required to finish a task (individual users)
 Number of tasks executed per time (throughput)
 Less time to finish implies better performance
Performance = 1 / Execution Time
 If X is n times faster than Y, then
3
PerformanceX
ExecutionTimeY
n

PerformanceY
ExecutionTimeX
CPU Execution Time
 How to measure the execution time?
 Almost all modern computers are based on a clock
 The clock is a periodic square wave with known
period (cycle time)
 Period = 1 / Frequency
one clock cycle
• The base unit in measuring time is the cycle time ,
thus
Time = cycles * cycle time
Time = cycles / clock frequency
4
CPU Execution Time
 Time required to execute a program is essentially
the time required to execute its instructions!
Time = #instructions x cycle time
 However, not all instructions take the same time!
 One way to think about execution time is that it
equals the number of instructions executed multiplied
by the average time per instruction
Time = #instructions x average cycles per instruction x cycle time
Time = IC x CPI x CC
5
CPU Execution Time
 The average CPI is computed by
Effective CPI =

N
k 1
ICk  CPI k
IC
 Where



Where ICk is the number of instructions of class k
executed
CPIk is the number of clock cycles per instruction for
that instruction class
N is the number of instruction classes
 Note: The overall effective CPI varies by instruction mix –
a measure of the dynamic frequency of instructions across
one or many programs
6
The Performance Equation
Execution Time = IC  CPI x CC
Execution Time = IC  CPI / CR
Performance = 1 / Execution Time
 Notes
 Three key factors for performance: IC, CPI, and CC
 CC: The clock rate is usually given
 IC: Overall instruction count (executed instructions)
by using profilers/ simulators
 CPI: varies by instruction type and ISA
implementation
7
The Performance Equation
 Example 1. In a certain program 1000 instructions
were executed on CPU running at 1 GHz. If the
instruction counts and CPI for each class are given
below, how long does it take to execute the program?
Instruction
Class
Instruction
Count
Class CPI
1
200
2
2
300
3
3
500
1
Effective CPI = (200x2+300x3+500x1)/1000 = 1.8
Time = 1000 x 1.8 x 1 ns = 1.8 us
8
The Performance Equation
 Example 2. Suppose the computer A has clock cycle
of 250 ps and CPI 2.0 for some program, and
computer B has clock cycle time of 500 ps and CPI of
1.2 for the same program, then which computer is
faster ?
TimeA = IC x 2 x 250 ps = 500 IC ps
TimeB = IC x 1.2 x 500 ps = 600 IC ps
PerformanceA
TimeB
600 IC
-------------------- = ---------- = --------- = 1.2
PerformanceB
TimeA
500 IC
9
Computer A is 1.2 faster than B
The Performance Equation
 Example 3. A certain processor that has four instruction classes is
to be modified using different approaches. The details of the program
used in evaluating different approaches are given in the table below.

What is the effective CPI for
 The original processor
 Approach 1. A cache is added and it reduces the average load time to 2 cycles.
 Approach 2. A branch prediction scheme is used and it cuts the branch time
by 1 cycle.
 Approach 3. A second ALU is added to execute two ALU instructions at once.
Original
10
App1
App2
App3
Class
Frequency
Class CPI
CPIk x F
CPIk x F
CPIk x F
CPIk x F
ALU
50%
1
0.5
0.5
0.5
0.25
Load
20%
5
1.0
0.4
1.0
1.0
Store
10%
3
0.3
0.3
0.3
0.3
Branch
20%
2
0.4
0.4
0.2
0.4
Effective CPI
2.2
1.6
2.0
1.95
Speed up
1.375
1.10
1.128
Determinants of Performance
Execution Time = IC  CPI x CC
IC
CPI
Algorithm
X
X
Programming
Language
X
X
Compiler
X
X
ISA
X
X
X
X
X
Processor
Organization
Technology
11
CC
X
SPEC Benchmark
 What programs can be used to evaluate
different computers? Can we cheat? Need a
standard!
 SPEC Benchmark
 Standard Performance Evaluation Corp (SPEC)
 Programs used to measure performance (CPU,
Web, I/O…)
 Typical actual workloads
 SPEC CPU2006




Elapsed time to execute a selection of programs
Negligible I/O, so focuses on CPU performance
Summarize as geometric mean of performance ratios
CINT2006 (integer) and CFP2006 (floating-point)
n
12
Geometric Mean =
n
 Execution Time Ratio
i 1
i
SPEC Benchmark
CINT2006 for Opteron X4 2356
IC×109
CPI
Tc (ns)
Exec time
Ref time
SPECratio
Interpreted string processing
2,118
0.75
0.40
637
9,777
15.3
bzip2
Block-sorting compression
2,389
0.85
0.40
817
9,650
11.8
gcc
GNU C Compiler
1,050
1.72
0.47
24
8,050
11.1
mcf
Combinatorial optimization
336
10.00
0.40
1,345
9,120
6.8
go
Go game (AI)
1,658
1.09
0.40
721
10,490
14.6
hmmer
Search gene sequence
2,783
0.80
0.40
890
9,330
10.5
sjeng
Chess game (AI)
2,176
0.96
0.48
37
12,100
14.5
libquantum
Quantum computer simulation
1,623
1.61
0.40
1,047
20,720
19.8
h264avc
Video compression
3,102
0.80
0.40
993
22,130
22.3
omnetpp
Discrete event simulation
587
2.94
0.40
690
6,250
9.1
astar
Games/path finding
1,082
1.79
0.40
773
7,020
9.1
xalancbmk
XML parsing
1,058
2.70
0.40
1,143
6,900
6.0
Name
Description
perl
Geometric mean
11.7
High cache miss rates
13
SPEC Benchmark
CINT2000 Results for Various Processors
Pentium 3
3400
CINT2000
Pentium 4
2900
P4 Extreme
2400
Xeon
Athlon
1900
Athlon 64
1400
Opteron
Pmac G5
900
Athlon FX (DC)
14
Clock Speed (GHz)
4
3.5
3
2.5
2
1.5
1
0.5
400
Core Duo
Core 2 Duo
SPEC Benchmark
CFP2000 Results for Various Processors
3200
Pentium 3
Pentium 4
2700
CFP2000
P4 Extreme
2200
Xeon
Athlon
1700
Athlon 64
Opteron
1200
Pmac G5
700
Athlon FX (DC)
Core Duo
15
Clock Speed (GHz)
3.5
3
2.5
2
1.5
1
0.5
200
Core 2 Duo
Other Performance Factors
 Power consumption is one factor in evaluating
performance
 This is specifically important in the embedded market
where battery life is important (and passive cooling)
16
Examples
 Example 4. given a program with 106 instructions with the
following mix: 10% class A, 20% class B, 50% class C, and
20% class D. If this program is executed on two different
processors with the specifications given below, then
Processor
CR
(GHz)
CPI Class A
CPI Class B
CPI Class C
CPI Class D
1
1.5
1
2
3
4
2
2
2
2
2
2
 What
is the effective CPI for the program for each
implementation?
 Which implementation is faster?
 What is the speedup?
17
Examples
 Example 5. The information for some program that is
executed on some processor is given below. If the
processor is modified such that the CPI for Class 2
instructions is reduced to 2, then would it be beneficial to
adopt this modification if this
 modification requires increasing the clock cycle by 10%?
 modification does not affect the clock cycle but requires twice
the amount of power to execute the program
18
Classi
CPIi
Frequencyi
1
2
0.3
2
5
0.2
3
3
0.5