The Forgotten Factor: FACTS
on Performance Evaluation
and its Dependence on Workloads
Dror Feitelson
Hebrew University
Performance Evaluation
• In system design
– Selection of algorithms
– Setting parameter values
• In procurement decisions
– Value for money
– Meet usage goals
• For capacity planing
The Good Old Days…
• The skies were
blue
• The simulation
results were
conclusive
• Our scheme
was better than
theirs
Feitelson & Jette, JSSPP 1997
But in their papers,
Their scheme was better than ours!
How could they be so wrong?
Performance evaluation depends on:
• The system’s design
(What we teach in algorithms and data structures)
• Its implementation
(What we teach in programming courses)
• The workload to which it is subjected
• The metric used in the evaluation
• Interactions between these factors
Performance evaluation depends on:
• The system’s design
(What we teach in algorithms and data structures)
• Its implementation
(What we teach in programming courses)
• The workload to which it is subjected
• The metric used in the evaluation
• Interactions between these factors
Outline for Today
• Three examples of how workloads affect
performance evaluation
• Workload modeling
• Research agenda
In the context of parallel job scheduling
Example #1
Gang Scheduling and
Job Size Distribution
Gang What?!?
Time slicing parallel jobs with coordinated
context switching
Ousterhout
matrix
Ousterhout, ICDCS 1982
Gang What?!?
Time slicing parallel jobs with coordinated
context switching
Ousterhout
matrix
Optimization:
Alternative
scheduling
Ousterhout, ICDCS 1982
Packing Jobs
Use a buddy system for allocating processors
Feitelson & Rudolph, Computer 1990
Packing Jobs
Use a buddy system for allocating processors
Packing Jobs
Use a buddy system for allocating processors
Packing Jobs
Use a buddy system for allocating processors
Packing Jobs
Use a buddy system for allocating processors
The Question:
• The buddy system leads to internal
fragmentation
• But it also improves the chances of
alternative scheduling, because processors
are allocated in predefined groups
Which effect dominates the other?
The Answer (part 1):
Feitelson & Rudolph, JPDC 1996
The Answer (part 2):
The Answer (part 2):
The Answer (part 2):
The Answer (part 2):
•
•
•
•
Many small jobs
Many sequential jobs
Many power of two jobs
Practically no jobs use full machine
Conclusion: buddy system should work well
Verification
Feitelson, JSSPP 1996
Example #2
Parallel Job Scheduling
and Job Scaling
Variable Partitioning
• Each job gets a dedicated partition for the
duration of its execution
• Resembles 2D bin packing
• Packing large jobs first should lead to better
performance
• But what about correlation of size and
runtime?
“Scan” Algorithm
• Keep jobs in separate queues according to
size (sizes are powers of 2)
• Serve the queues Round Robin, scheduling
all jobs from each queue (they pack
perfectly)
• Assuming constant work model, large jobs
only block the machine for a short time
Krueger et al., IEEE TPDS 1994
Scaling Models
• Constant work
– Parallelism for speedup: Amdahl’s Law
– Large first  SJF
• Constant time
– Size and runtime are uncorrelated
• Memory bound
– Large first  LJF
– Full-size jobs lead to blockout
Worley, SIAM JSSC 1990
The Data
Data: SDSC Paragon, 1995/6
The Data
Data: SDSC Paragon, 1995/6
The Data
Data: SDSC Paragon, 1995/6
Conclusion
• Parallelism used for better results, not for
faster results
• Constant work model is unrealistic
• Memory bound model is reasonable
• Scan algorithm will probably not perform
well in practice
Example #3
Backfilling and
User Runtime Estimation
Backfilling
• Variable partitioning can suffer from
external fragmentation
• Backfilling optimization: move jobs
forward to fill in holes in the schedule
• Requires knowledge of expected job
runtimes
Variants
• EASY backfilling
Make reservation for first queued job
• Conservative backfilling
Make reservation for all queued jobs
User Runtime Estimates
• Lower estimates improve chance of
backfilling and better response time
• Too low estimates run the risk of having the
job killed
• So estimates should be accurate, right?
They Aren’t
Mu’alem & Feitelson, IEEE TPDS 2001
Surprising Consequences
• Inaccurate estimates actually lead to
improved performance
• Performance evaluation results may depend
on the accuracy of runtime estimates
– Example: EASY vs. conservative
– Using different workloads
– And different metrics
EASY vs. Conservative
Using CTC SP2 workload
EASY vs. Conservative
Using Jann workload model
EASY vs. Conservative
Using Feitelson workload model
Conflicting Results Explained
•
•
•
•
Jann uses accurate runtime estimates
This leads to a tighter schedule
EASY is not affected too much
Conservative manages less backfilling of long
jobs, because respects more reservations
Conservative is bad for the long jobs
Good for short ones that are respected
Conservative
EASY
Conflicting Results Explained
• Response time sensitive to long jobs, which
favor EASY
• Slowdown sensitive to short jobs, which
favor conservative
• All this does not happen at CTC, because
estimates are so loose that backfill can
occur even under conservative
Verification
Run CTC workload with accurate estimates
But What About My Model?
Simply does not
have such small
long jobs
Workload Modeling
No Data
• Innovative unprecedented systems
– Wireless
– Hand-held
• Use an educated guess
– Self similarity
– Heavy tails
– Zipf distribution
Serendipitous Data
• Data may be collected for various reasons
–
–
–
–
Accounting logs
Audit logs
Debugging logs
Just-so logs
• Can lead to wealth of information
NASA Ames iPSC/860 log
42050 jobs from Oct-Dec 1993
user
user4
user4
user42
user41
sysadmin
user4
sysadmin
user41
job nodes runtime date
time
cmd8 32
70 11/10/93 10:13:17
cmd8 32
70 11/10/93 10:19:30
nqs450 32 3300 11/10/93 10:22:07
cmd342 4
54 11/10/93 10:22:37
pwd
1
6 11/10/93 10:22:42
cmd8 32
60 11/10/93 10:25:42
pwd
1
3 11/10/93 10:30:43
cmd342 4 126 11/10/93 10:31:32
Feitelson & Nitzberg, JSSPP 1995
Distribution of Job Sizes
Distribution of Job Sizes
Distribution of Resource Use
Distribution of Resource Use
Degree of Multiprogramming
System Utilization
Job Arrivals
Arriving Job Sizes
Distribution of Interarrival Times
Distribution of Runtimes
Job Scaling
User Activity
Repeated Execution
Application Moldability
Distribution of Run Lengths
Predictability in Repeated Runs
Research Agenda
The Needs
•
•
•
•
•
New systems tend to be more complex
Differences tend to be finer
Evaluations require more detailed data
Getting more data requires more work
Important areas:
– Internal structure of applications
– User behavior
Generic Application Model
• Iterations of
– Compute
• granularity
• Memory working set / locality
– I/O
• Interprocess locality
– Communicate
• Pattern, volume
• Option of phases with
different patterns of iterations
compute
I/O
communicate
Consequences
• Model the interaction of the application
with the system
– Support for communication pattern
– Availability of memory
Application attributes depend on system
Effect of multi-resource schedulers
Missing Data
• There has been some work on the
characterization of specific applications
• There has been no work on the distribution
of application types in a complete workload
– Distribution of granularities
– Distribution of working set sizes
– Distribution of communication patterns
Effect of Users
• Workload is generated by users
• Human users do not behave like a random
sampling process
– Feedback based on system performance
– Repetitive working patterns
Feedback
• User population is finite
• Users back off when performance is
inadequate
Negative feedback
Better system stability
• Need to explicitly model this behavior
Locality of Sampling
• Users display different levels of activity at
different times
• At any given time, only a small subset of
users is active
• These users repeatedly do the same thing
• Workload observed by system is not a
random sample from long-term distribution
Final Words…
We like to think
that we design
systems based
on solid
foundations…
But beware:
the foundations
might be
unbased
assumptions!
Computer Systems are Complex
We should have more “science” in computer
science:
• Run experiments under different conditions
• Make measurements and observations
• Make predictions and verify them
Acknowledgements
• Students: Ahuva Mu’alem, David Talby,
Uri Lublin
• Larry Rudolph / MIT
• Data in Parallel Workloads Archive
–
–
–
–
–
Joefon Jann / IBM
CTC SP2 log
SDSC Paragon log
SDSC SP2 log
NASA iPSC/860 log