CSCI1600: Embedded and
Real Time Software
Lecture 19: Queuing Theory
Steven Reiss, Fall 2015
Queuing Theory History
 Agner Krarup Erlang (1878-1929)
 Analysis of telephone networks (1908-1922)
 Why does this involve queues?
Analysis of Telelphone Networks
 Issues
 How many operators are needed
 How many circuits are needed
Queuing Theory
 A branch of probability theory studying queues
Arriving
Jobs
 Applications
 Computer systems
 Communication networks
 Call centers
 Transportation
 …
Server
Departing
Jobs
Parameters: Arrivals
Arriving
Jobs
 Arrival rate λ
 How fast jobs are arriving
 Regular or random
 Random = Poisson process
 With some distribution
Server
Departing
Jobs
Parameters: service
Arriving
Jobs
Server
Departing
Jobs
 Service demand S – amount of service the job needs
 Measured in various terms (time, size, …)
 Regular or random (with a given distribution)
 Service capacity C – how fast server process jobs
 Service rate μ = C/S
 Service time = S/C = 1/μ
Parameter: Service
Arriving
Jobs
Server 1
Departing
Jobs
Server m
 Multiple symmetric servers
 m parallel servers constitute a single serving station
 Each has capacity C
 Overall capacity depends on number of serviced jobs
 C(N) = min(N,m)C
Parameters: Queue
Arriving
Jobs
Server
Departing
Jobs
 Scheduling discipline
 First come, first served (FCFS, FIFO)
 Priority queue (Shortest in, first out)
 OS I/O scheduler
 Preemptive vs. nonpreemptive servicing
 Preemptive servicing can be interrupted by higher priority job
 Linux real-time thread scheduling
Statistical Basis for Queuing Theory
 Arrival probability distribution
 Service probability distribution
 Determine various properties of the system
 Average queue size
 Maximum expected queue size
 Average wait time
 Maximum expected wait time
Poisson Arrival
 A completely random arrival process
 The number of jobs is very large
 Jobs are independent of each other
 Job arrivals are not simultaneous
 λ: the average arrival rate
 Examples
 Phone calls to a switchboard
 Radioactive decay
 Customers going to checkout
 Trains needing switches thrown
Poisson Process
 Represented by Poisson distribution
 Probablity of seeing i jobs during the time interval T
 P[n(t) = i] = (λT)i/i! * e-λT
PDF
CDF
Poisson Process Properties
 Memoryless
 The number of arrivals in the time interval (T;T+Δt)
 Does not depend on the number before T
 Superposition
 Combination of m independent Poisson processes with
λ1,λ2,…λm is a Poisson process with λ= λ1+λ2+…+λm
 Interval between job arrivals
 Is distributed exponentially fx(x) = λe-λx
Exponential Service Distribution
 PDF: fs(t) = μe-μt where μ is the service rate
 CDF: Fs(t) = 1 - μe-μt
 Mean = Std. Deviation = 1/μ = service time
Other Distributions
 Erlang
 The job visits an exponential server k times
 Hyperexponential
 k types of jobs, each has probability πi
 Each type of job has its own service time
 Each type of jobs passed to its own server
 General
 With known mean and standard deviation
A/S/n Notation
 Proposed by D. Kendeall in 1953
 Arrival process (A) – message arrival distribution
 Server process (S) – servicing time distribution
 Number of servers (n)
 Distribution lables
 M : exponential (memoryless, Poisson)
 D : deterministic (constant arrival/processing times)
 G : general
 Ek : k-stage Erlang
 Hk : k-stage Hyperexponential
Examples
 M/M/1
 One server for jobs that arrive and are serviced by Poisson
process
 System with many independent jobs sources can be
approximated this way
 M/D/n
 Arrival process is Poisson, service is deterministic
 There are n servers
 G/G/n
 General system with arbitrary arrival and service times
Goals of Queuing Theory
 Predict the performance of the system in steady state
 Not startup or shut down
 Assumptions
 The population of jobs is infinite
 The size of the queue is infinite (can be bounded)
 Specific distributions for A and S known
Performance Metrics
 W: waiting time (queuing delay)
 Time between receipt of a job and start of processing
 Depends on the load on the server
 S: service time (processing delay)
 The actual processing time of the job
 Depends on the serving capacity
 T: response time
 T=W+S
 Throughput
 Average number of jobs per time unit
Little’s Law
 Number of jobs in the system N(t)
 N = λT
 λ : the mean arrival rate
 T : the mean service time
 Queue size Q
 Q = max(0,N – m)
 Waiting time in the Queue W
 Q = λW
Little’s Law Example
 Clothing store
 Customers arrive 10/hour and stay 30 minutes
 λ = 10, T = 0.5
 Average number of customers in the store N = 10*0.5 = 5
 Register line subsystem in the store
 Average register queue is 2 people
 The wait time W = Q/λ = 2/10 = 0.2 hours
Other Metrics
 Traffic intensity p
 Single server: p = arrival rate * service time = λ/μ
 m servers: p = λ/(mμ)
 p is dimensionless, measured in Erlangs
 If p > 1, then more jobs arrive than the system can handle
 p < 1 needed to prevent overload
 Utilization factor
 U = min(p,1)
Analytic Solutions
 Some queuing models can be solved analytically
 M/M/1 for example
 Other variants
M/M/1 Performance
 Average # jobs in the system N = p/(1-p)
 Average queue size Q = p2/(1-p)
 Average wait time W =
𝑝
μ 1−𝑝
 Average response time T =
1
μ 1−𝑝
M/M/1 Example
 Post office
 Customers arrive 10 per hour (λ=10)
 Clerk can handle 16 customer per hour (μ = 16)
 𝑝=
𝑙
𝑢
=
10
16
= 0.625
 Result
 Average number of customers N = 1.667
 Average queue size Q = 1.042
 Average wait time W = 0.1042 hours (6.25 minutes)
 Average response time T = 10 minutes
How to Fix Overload
 How can we reduce job delays
 Obvious:
 Shorten μ (speed up the code, profile, optimize)
 Lengthen λ (decrease the load, make less bursty)
 Move to M/M/m
 How many servers do you want
 This can be solved as well
Problems
 Classical queuing theory is too restrictive
 Infinite number of jobs
 Only certain distributions are handled
 Job routine is not allowed
 Some limitations can be alleviated
 But calculations and the math involved are complex
 Provides insufficient information
Solving Queuing Problems
 Use a computer representation of the system
 Instead of a mathematical model
 Use simulation to learn the properties
 This is what is generally used
 MATLAB SimEvents
 OMNET
 Can construct arbitrary models and run them
Homework
 For next week (10/26)
 Present you project to the class
 Plans, progress to date
 Project model
 Tasks and model(s) for each task
 Hand this in
 Can be part of presentation