# QUEUE

Document Sample

```					Queuing Theory
Web Performance

We talked about the importance of
performance in web applications
It is not easy to separate the different
components of response time
It is important to be able to make
predictions from information that is easy to
gather.
Approaches to performance analysis

Try new configuration and observe results
Scale up from prior results
Arrival rate .5, mean 43, max 627
Arrival rate .95, mean 1859, max 4285
Arrival rate .99, mean 2583, max 5068
Develop queuing theoretic model
Run a simulation
Predicting response from past
experience
Queue Model
Multiserver queue Model
Single web server forwards requests to cluster
Multiple Single-server queues
DNS rotation for cluster
A Quick Probability Refresher
 A random variable, X, can take on a number of different
possible values
 Example: the number of students who came to office hours is a
random variable with values 1,2,3,…
 Each time we observe (or sample) the random variable,
it may take on a different value
 A random variable takes on each of these values with a
specified probability
 Example: X = {0, 1, 2, 3, 4}
 P[X=0] = .1, P[X=1] = .2, P[X=2] = .4, P[X=3] = .1, P[X=4] = .2
 The sum of the probabilities of all values equals 1
 all values P[X=value] = 1
A Quick Probability Refresher
 Example
 Suppose we throw two dice and the random variable, X, is the
sum of the two dice
 Possible values of X are {2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12}
 P[X=2] = P[X=12] = 1/36
 P[X=3] = P[X=11] = 2/36
 P[X=4] = P[X=10] = 3/36
 P[X=5] = P[X=9] = 4/36
 P[X=6] = P[X=8] = 5/36
 P[X=7] = 6/36

12
Note:

i=2
P[X=i] = 1
A Quick Probability Refresher
 A probability distribution function matches each
possible value of a random variable with its
associated probability
0.180
0.160
0.140
0.120
0.100
0.080                                                  P[X]
0.060
0.040
0.020
0.000
2   3   4   5   6   7   8   9   10   11   12
A Quick Probability Refresher

The cumulative distribution function of a
random variable, X, is defined by
CDF: P[X  x] = all y=x P[x=y]

1
0.9
0.8
0.7
0.6
0.5
CDF
0.4
0.3
0.2
0.1
0
2   4   6   8   10   12
A Quick Probability Refresher
 Expected Value
Can be thought of a “long term average” of observing
the random variable a large number of times

E[X] = x =       Value * P[X = value]
All possible
values of x

 Example: dice
E[X] = 2*1/36 + 3*2/36 + 4*3/36 + 5*4/36 + 6*5/36 +
7*6/36 + 8*5/36 + 9*4/36 + 10*3/36 + 11*2/36 +
12*1/36
A Quick Probability Refresher
 Average vs. Expected Value
 Short term average
 Suppose a random variable X is sampled N times
 Let ni = # of X = i was observed
 Average of samples
• = (n0*0 + n1*1 + n2*2 + n3*3 + …)/N
• = n0/N*0 + n1/N*1 + n2/N*2 + n3/N*3 + …
 As N  , the ratio ni/N becomes pi
 Thus, E[X]
 = limN [n0/N*0 + n1/N*1 + n2/N*2 + n3/N*3 + …]
 = p0*0 + p1*1 + p2*2 + p3*3 + …
 = i=0 i * pi
A Quick Probability Refresher
 Continuous Random Variables
 In many cases, a random variable takes a value drawn from a
continuous interval
 Ex: processing time for a packet may be any real value [0, )
 The distribution of possible values a continuous random variable
can take is given by a probability density function, F(x)
 P(a  x  b) = ab F(x)dx = bi=a P(x = i)
 E[x] = - xF(x)dx =  i * P(x = i)
Exponential Distribution
 The cumulative distribution function F(x)
and probability density function f(x) are:
1               F ( x ) = 1 - e -l x                         1        f ( x ) = l e -l x       x³0
0.8                                                          0.8
F(x) = 1 - exp(-x)

f(x) = exp(-x)
0.6                                                          0.6

0.4                                                          0.4

0.2                                                          0.2

0                                                            0
0      1        2         3       4                          0           1            2     3   4
x                                                                     x
In queuing theory, we often assume the service time follows an exponential distribution.
The service time corresponds to the packet transmission time and is proportional to
the packet length.
When standard deviation is equal to mean, we estimate it to be exponential
Poisson Distribution
Poisson Distribution

 The random variable X has a Poisson distribution with mean ,
if for non-negative integers i:
 P[X = i] = (ie-)/i!
 Represents the number of arrivals in a time unit given an
exponential arrival process
 Facts
 Discrete distribution
 E[X] = 
Poisson Point Process
 Definition
 A Poisson point process with parameter 
 A point process with interpoint times that are independent and
exponentially distributed with parameter .

Mean interarrival time = 1/, with
exponential distribution
Poisson Point Process

 Equivalently
 The number of points in disjoint intervals are independent, and the
number of points in an interval of length t has a Poisson
distribution with mean t

Shown are three disjoint intervals. For a Poisson point process, the number
of points in each interval has a Poisson distribution.
Poisson Point Process

 Exercise
Given a Poisson point process with rate  = 0.4,
what is the probability of NO arrivals in an interval
of length 5?
Mean = 1/          a                     b=a+5

Length 5 interval

Try to answer two ways, using two equivalent descriptions of a Poisson
process
Poisson Point Process

X = time from a until next point
X

N = number of points in
interval
(Poisson with mean 5)

Solution 1: P[X > 5] = e-5 = 0.1353

Solution 2: P[N = 0] = e-5 = 0.1353
(remember: P[N = i] = (5)i * (e-5) / i!, for i = 0)
Basic Queueing Theory
 Elementary notions
 Things arrive at a queue according to some probability
distribution
 Things leave a queue according to a second probability
distribution
 Averaged over time
 Things arriving and things leaving must be equal
 Or the queue length will grow without bound
 Convenient to express probability distributions as average rates
Little’s Law
 Goal
 Estimate relevant values
 Average number of customers in the system
• The number of customers either waiting in queue or receiving service
 Average delay per customer
• The time a customer spends waiting plus the service time
 In terms of known values
 Customer arrival rate
• The number of customers entering the system per unit time
 Customer service rate
• The number of customers the system serves per unit time
Little’s Law

For any box with something steady flowing
through it

Allows us to express the
natural idea that
crowded systems (large
Mean amount in box                                N) are associated with
long customer delays
(average number of           N = T
things in the queue)
Mean throughput               Mean time spent in box
(rate at which things enter and   (average time spent by a thing
leave the queue)                  in the queue)
Little’s Law

N = T
Mean amount in box                                      Mean time spent in box
Mean throughput
 Example
 Suppose you arrive at a busy restaurant in a major city
 Some people are waiting in line, while other are already seated (i.e., being served)
 You want to estimate how long you will have to wait to be seated if you join the end of
the line
 Do you apply Little’s Law? If so
   What is the box?
   What is N?
   What is ?
   What is T?
Little’s Law

N = T
Mean amount in box                                      Mean time spent in box
Mean throughput
 Box
 Include the people seated (i.e., being served)
 Do not include the people waiting in line
 Let N = the number of people seated (say 200)
 Let T = mean amount of time a person stays seated (say 90 min)
 Conclusion
 Throughput = 200/90 = 2.22 persons per minute
 Wait time
 If 100 people are waiting, you could estimate that you will need to wait 100/2.22 = 45
min
Little’s Law

622 Mbps

 Suppose ATM streams are multiplexed at an output link with
speed 622 Mbps
 Question
 If 200 cells are queued on average, what is the average time in
queue?
 T = N/
 T = 200 * 53 * 8 / 622Mbps
 T = 0.136 ms
Queue Statistics

Arrival rate = load - Poisson - l
Packet length Distribution - exponential
E(m) expected value (mean) of random variable
Ts=Mean service time, Tq=Mean queue time
Standard Deviation of service time s T   s

Link utilization   r = l Ts
Fixed Packets

Average transmission time
  = average packets serviced/sec = capacity
1
Ts =
m
l
r = = utilization £ 1, l < m
m                             n
Buffer
Arriving                              Departing Packets
Packets

l                            m
Practical Example

=Tr
=Ts
Little’s Law
Applies to systems where no jobs are
lost or created
Arrival rate = arrivals/total time=N/Tt
N
J = å Tn
n= 0
Mean time in system = J/N
Mean number in system=
J  N J
= × = Arrival rate × mean time in system
Tt Tt N
q = lTq
Examine time up to D6=19

T1=3
Arrival Rate = N/Tt=6/19=.31
T2=7
Mean Time in system = J/N = 38/6=6.3
T3=10
Mean number in system = J/Tt=38/19=2
T4=6
= (J/N)*(N/Tt)=6.3*.31=2
T5=6
=Tq
T6=6
J=38
Throughput

Throughput=utilization/service time
Throughput =/Ts
For =.5 and Ts=1ms
Throughput is 500 packets/sec
Little’s Law Examples N = rT
 How long to answer email
 50 messages/day
 150 messages in Inbox
 Hospital
 2.5 days stay
 5 new cases/day
 ~12.5 in the system
 Average Time in Hospital
 102.5 average patients
 Discharge 67.5 patients/day
 ~1.5 days in hospital
 Customers in store
 10/hour arrival rate
 Stay on average 0.5 hours
 ~5 in store
Queueing Theory
 Mathematical study of queues
 Enables analysis of several related processes
 Can calculate several steady-state performance
measures
 average waiting time
 expected number waiting
 probability of a given number of items in the queue
 etc.
 Can model several different types of systems
 Different queueing disciplines
 Different arrival, service, and departure probability distributions
 etc.
Kendall Notation
 A/S/NS/B/K/SD
A,S=Interarrival time, service time distribution
 M = Exponential
 Ek = Erlang
 Hk = Hyperexponential
 D = Deterministic
NS=Number of servers
B=Number of Buffers
K=Population size
SD=Service Discipline
 FCFS,FCLS…
Kendall Notation
 A/S/NS/B/K/SD
 Defaults
B=  ,                 number of buffers
K=  ,                 population size
SD=FCFS                service discipline
 M/M/1 = M/M/1/  /  /FCFS
exponential arrival rate and service time
1 server
infinite buffers and infinite population
FCFS queue
M/M/1 queue

E(k) = expected arrivals in T seconds = lT
Average time between arrivals
¥            ¥
- lt
E(t ) = ò tf (t )dt = ò tle          dt = 1 l
0            0

Where f (t ) is the probability density function
for an exponential distribution
Markov Models - analyze state n
(n buffers are occupied)
• n+1         departure
Buffer
Occupancy             •n                      •n

• n-1         arrival

t                       Dt
l                       l       l

0       1   ...       n-1       n       n+1

m                       m       m
Probability of being in state n
None arrived or departed     one arrived and one departed

None arrived, one departed

one arrived, none departed
Markov Chains
l                           l        l

0          1      ...     n-1        n        n+1

m                           m        m
Rate leaving n = Pn (l + m )
Rate arriving n = lPn -1 + mPn +1
Steady State Pn (l + m ) = lPn -1 + mPn +1
State 0 lP0 = mP1
Substituting Utilization
l
P1 = P0 = rP0
m
Rate leaving n = rate entering n
Pn ( l + m) = lPn-1 + mPn +1
P1 ( l + m) = lP0 + mP2
mP2 = ( l + m)P1 - lP0
l         l
P2 = P1 + P1 - P0 = P1 ( r + 1) - rP0
m         m
Substituting P1

P2 = rP0 (r + 1) - rP0
= r P0 + rP0 - rP0 = r P0
2                                 2

Pn = r P0   n

• Higher states have decreasing probability
• Higher utilization causes higher probability
of higher states

¥            ¥               P0
¥
å Pn = 1 = å r P0 = P0 å r =
n              n

n= 0       n= 0        n= 0  1- r
P0
1=        ® P0 = 1 - r
1- r
Pn = (1 - r )r n
l
Queue determined by     r=
m
Single Server

(Ts)/Ts>>1           (Ts)/Ts=1   Ts=0
Example

Given things that you can easily measure
 Arrival rate of 1000 requests/sec
Service capacity of 1100 requests/sec
Find                l 1000
Utilization     r= =     = 0.91
m 1100
Probability of having 4 buffers filled

P4 = (1 - r )r =.062
4

P1 =.082, P2 =.075, P3 =.068, P5 =.056
Example
Mean number of items in the system
r
r = E(n) =        = 9.99
1- r
With infinite buffers to determine delay > bound
P(n > 12) = r12+1 = .28
With 12 fixed buffers, loss probability
(1- r)r12
P12 =           = .04
1- r12+1

Pn = .11,.10,.09,.09,.08,.07,.07,.06,.05,.05,.04
Examples

 Web Site with 100 Clients and 1 server.
 Service time .6sec and = (exponential)
 What is average response time at 20 queries/min=1q/3sec?
 Use M/M/1 model
 =Ts=(1/3)(.6)=0.2
 Tr=Ts/(1-)=.6/.8=.75sec
 If 1.5 seconds is too long, what utilization is allowable (90%
of responses are less than 1.5sec)
 mTr(r)=Tr*ln(100/(100-r))
 mTr(90)=Tr*ln(100/10)=[Ts/(1-)]*2.3=1.5 sec
 2.3 Ts=(1-)1.5 sec, 1.5  = 1.5-2.3*.6 , =.08
 The utilization must actually be reduced
Multi-server System with separate
queues (DNS Rotation)
 five processors (5 M/M/1), average service time= 0.1
sec.
 Assume that the standard deviation of service time is
observed to be 0.094 sec (exponential service time).
 Web page hits= 40 per second.
 Separate Queue Approach
 If processes are evenly distributed among the
processors, then the load for each processor is
 40/5 = 8 requests per second. Thus,
  =  Ts = 8  0.1 = 0.8
 The residence time is then easily calculated:
 tr = Ts/(1- )=.1/.2=.5 seconds
Single Queue - Multiple server
 (M/M/5) aggregate arrival rate of 40 processes per second.
Utilization is still =( Ts /5) = 0.8
 To calculate the residence time first calculate Erlang C function,
table lookup,  =0.8 for 5 servers C = 0.554
 Tr = (C/N)(Ts/(1- ))+ Ts= ((0.544)(0.1))/(5(1- 0.8))+ (0.1) =
0.1544
 So the use of multiserver single queue has reduced average
residence time from 0.5 sec (previous slide) down to 0.1544
sec,
 which is greater than a factor of 3.
 If we look at just the waiting time, the single queue case is
0.0544 seconds compared to 0.4 seconds, for multiple queues
 which is a factor of 7.
Another Example
 Arrival rate
 = 125 requests/sec
 Service rate
=1/0.002 = 500 requests/sec
 Gateway utilization
= / =0.25
 Probability of n packets in gateway =
(1- ) n =0.75*(.25)n
 Mean time spent in web server =
Ts/(1- )=.002/(1-.25)=2.66ms
Simulation
Identify Performance Problems
Problems may only occur under high load
Benchmark Web Components
Deployment decisions
Evaluate new features
Capacity Planning
Determine network, memory, disk and
clustering needs
 Based on the results of
numerous studies                Category   Parameter
 Key properties                  Protocol   Request Method
 HTTP Message                            Response Code
Characteristics              Resource   Content type
 Several request methods              Resource size
and response codes                   Response size
 Resource Characteristics                Popularity
 Diverse content-type,                Modification freqency
size, popularity, and                Temporal Locality
modification frequency
# embedded resources
 User Behavior
Users      Session interarrival times
 User browsing habits
# clicks per session
significantly affect
Parameter Characterization

Associate each parameter with
quantitative values
Statistics
Mean, median, mode
OK for parameters that don’t vary much
Probability Distributions
Capture how a parameter varies over a wide range
of values
Probability Distribution
 Every random variable gives rise
to a probability distribution
 Probability Density Function
 Assigns a probability to every
interval of the real numbers
 Cumulative Distribution Function
 Describes the probability
distribution of a real-valued
random variable X
 F(x) = P(X <= x)
 The probability that a random
variable will be less than or
equal to x
 In the following slides, we will
show the CDF of commonly used
distributions
Poisson Distribution
• F(x) = (e-k)/k!
• Used to model the
time between
independent events
that happen at a
constant average rate

• The number of times a web server is accessed per
minute is a Poisson distribution
–   For instance, the number of edits per hour recorded on Wikipedia's Recent Changes page
follows an approximately Poisson distribution.
Exponential Distribution
• F(x) = e-x
• Used to model the
time until the next
occurrence of an
event in a Poisson
process

• Session interarrival times are exponential
– Time between the start of one user session and the start of the next
user session
Pareto Distribution
• F(x) = (x/a)-k
• k is shape, a is
minimum value for x
• Power law
• 80-20 rule
–   20% of the sample is responsible
for 80% of the results

• Response sizes, Resource sizes, Number of
embedded images, Request interarrival times
• Often used to model self-similar patterns
Probability Distributions in
Exponential    Session interarrival times
Pareto         Response Sizes
Resource Sizes
Number of embedded images
Request interarrival times
Lognormal      Response sizes
Resource sizes
Temporal Locality
Zipf-like      Resource Popularity
Probability Distribution Conversion

 Most languages have random number
library functions
 Uniform distribution
 Must convert from uniform distribution
to the chosen distribution

• Given: the cumulative distribution function, CDF,
of the chosen distribution
– 1. Generate a random number; call this number p
– 2. Compute x such that CDF(x) = p
• Determine the inverse of the CDF
– 3. x is the random number you use
Inverse of the CDF
For the exponential
distribution

```
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
 views: 8 posted: 8/15/2012 language: English pages: 63
How are you planning on using Docstoc?