Course Information
Andy Wang
CIS 5930
Computer Systems
Performance Analysis

Contact Information
• Andy Wang (awang@cs.fsu.edu)
• Office: 269 Love Building
• Office hours: after class
(also by appointments)
• Class website: http://www.cs.fsu.edu/~awang/courses/cis5930_s2015
2

Objectives
• Become exposed to common issues involved in evaluating software systems
• Learn statistical tools and experimental design techniques
• Gain experience in performance evaluation
3

What Will This Course
Teach You?
• Proper methods for designing and performing experiments on systems
• Proper methods for analyzing and presenting data gathered from such experiments
• Proper methods for critiquing experiments and data gathered
4

What Won’t This Course
Teach You?
• Basic systems software principles
• Systems software modeling
• Queueing theory
• Simulation techniques for systems software
5

Who Should Take
This Course
Well, everyone, but especially -
• Software developers
• Software researchers
• Software purchasers
• Software evaluators
6

Course Materials
• Lecture notes and papers (posted on the class website)
• Textbook
– The Art of Computer Systems
Performance Analysis, by Raj Jain

Grading
• Up to 5 assignments (take home exams) - 10% each
• Peer evaluation of projects by other groups - 10%
• Project – remaining grade
8

Project
• You need to develop a project in teams of two or three
• Goal:
– Publishable results

Suitable Subjects for Projects
• OSes or their components (file systems,
I/O subsystems, process handling, etc.)
• Compilers
• Databases
• Real-time applications
• Application packages
• Networks/networking systems
• The Web
10

Possible Projects
• Measure/simulate/compare systems
• Develop and validate a simulator
• Characterize workloads of a system
• Analyze the effects of various factors on a system
• Develop a software monitor to observe the performance of a system
11

Project Proposal
• Due on the 2 nd week
– All team members are required to participate
• 2-page written proposal
– System background and evaluation goals
– Metrics
– Parameters, factors, levels
– Evaluation technique
– Workloads
– Division of labors

Experimental Design
• Due on the 4 th week
– All team members are required to participate
• 2-page written design
– Brief background
– Experimental design
– Data analysis techniques
– Anticipated graphs

Project Presentation
• During the last week of the course
• 15-page (max) written paper due by the last lecture (double column, singlespace, 10-pt font)

Grading of Projects
Several criteria will be used:
• Proper design of the experiment
• Care and thoroughness of its execution
• Completeness of analysis
• Quality of data presentation
• Insight gained from experiment
Insight is the most important
15

Evaluation of Projects by Other Groups
• Submitted by each student individually
• Two 1-page critiques on other group’s proposed experiment
– Due by 5 th week
• Two 1-page critiques on other group’s results
– Due by final’s week
• Graded on basis of insight into strengths and flaws of each project
16