Documents
Resources
Learning Center
Upload
Plans & pricing Sign in
Sign Out

Capacity Planning for LAMP Architectures - Kitchen Soap

VIEWS: 0 PAGES: 37

									                 Capacity Planning for LAMP
                              Architectures
                              Web Builder 2.0 Las Vegas




John Allspaw
Manager, Operations
Flickr.com
                Capacity Planning for LAMP
                             Architectures

Capacity planning:

“the ability to make snap decisions to spend millions
   of dollars with not enough information”

                              - Kevin Murphy
                 Capacity Planning for LAMP
                              Architectures



                        It is NOT:

•   Performance tuning
•   Tips and tricks to be “scalable”
                Capacity Planning for LAMP
                             Architectures



                         It IS:

•   What comes after you’ve made it all “scalable”
•   Making sure that you have enough equipment to
    handle gradual and bursty traffic
Capacity Planning for LAMP
             Architectures




Questions to answer
                  Capacity Planning for LAMP
                               Architectures



How many of each server “class” should you add as
  you grow ?

      Hint: Don’t add too much (too much $$! Ahh!)
      Hint: Don’t add too little (too much traffic! Ahh!)
                       Capacity Planning for LAMP
                                    Architectures


How to make it easier to predict the future* ?

How to make it easier to justify those predictions ?

How to make it easier to predict the future…in the future ?!




*You can’t predict the future, but you can try.
                      Capacity Planning for LAMP
                                   Architectures

         The OLD way of doing things was easier

•   A.k.a. “web1.0”

•   Small number of content producers

•   Control over the content

•   Capacity was dictated by the demand for that static/small
    content

•   Even bbs/communities/ecommerce had relatively predictable
    growth
                 Capacity Planning for LAMP
                              Architectures

      Today’s way of doing things is harder/fun

•   No control over content (users have control)

•   No control over usage (users have control)
                 Capacity Planning for LAMP
                              Architectures

      Today’s way of doing things is harder/fun

•   Network effect, nonlinear growth (more
    users/content/contacts/activity mean >> usage)

•   Event-related growth (press, news event, social trend,
    etc. can affect usage and content)

Example: London bombing, tsunami, holidays, etc.
                Capacity Planning for LAMP
                             Architectures

       Considerations for “social” applications

•   User behavior should guide you with defining
    capacity metrics (not just server stats)

•   Usage can accelerate, not just grow
Capacity Planning for LAMP
             Architectures




How we do it at Flickr
                     Capacity Planning for LAMP
                                  Architectures

                         Gathering Usage

   Application-level information (users, photos, activity, etc.)

   Server-level information (cpu, disk I/O, memory, etc.)

   We tie the two together
                     Capacity Planning for LAMP
                                  Architectures

         BEFORE we start collecting server stats

   What resources are peak-driven ? (concurrent use)
    – Ex: photo processed/sec, pages/sec, images/sec, db qps


   What resources are permanently consumable ?
    – Ex: database space, storage (GB/day) etc.
                    Capacity Planning for LAMP
                                 Architectures

         BEFORE we start collecting server stats

   What is an average user:consumption ratio ? (example: user:
    photo)


   What is the high and low of ratios ?


   Is the average ratio changing over time ?
  Capacity Planning for LAMP
               Architectures




Non-linear growth
  Capacity Planning for LAMP
               Architectures




Non-linear growth
      Capacity Planning for LAMP
                   Architectures




Linear relationships, though
                   Capacity Planning for LAMP
                                Architectures

              Server and Network statistics

   Ganglia - (we love ganglia!)
    – Multicast-y goodness
    – SUPER simple to make a graph from any stat
    – Clustering


   Other custom rrdtool-based stuff

   MRTG
Capacity Planning for LAMP
             Architectures

           Photos uploaded/processed/min




           Avg processing time per minut




           Avg CPU per minute
                     Capacity Planning for LAMP
                                  Architectures

Gather and record statistics

   Accept the ‘observer effect’ (it’s worth it)

   Aggregate your stats across clusters

    – Stacked graphs
    – Totals and averages
Capacity Planning for LAMP
             Architectures
                Squid client request
                (24 hours)

                (Y axis is req/sec)
Capacity Planning for LAMP
             Architectures
                 Squid
                 LRU reference age
                 Over 24 hours

                 -Y axis is “days”
                 -So peak has 3.6hou
                       Capacity Planning for LAMP
                                    Architectures

Find the ceiling of each class/function/server

   Maximum allowable somethings

    – MySQL : queries/sec before slave lag sets in

    – Apache/php : page requests/sec before total meltdown

    – Squid/memcached : cache churn rate, request rate

    – Storage : disk I/O utilization, storage limit(!)

    – Etc.
                 Capacity Planning for LAMP
                              Architectures

Forget benchmarking, use real load
   – Make sure you have a easy mechanism to take servers in
     and out of production

   – Pull machines from a balanced pool during medium-level
     traffic (very carefully)

   – Watch and record
                       Capacity Planning for LAMP
                                    Architectures

Build the infrastructure to make it EASY to measure

Obvious things to help this:

   Load balancing
   Network segmentation
   Carve up functions into clusters
    – Don’t let a machine do more than one primary thing (if you can help it)



    this isn’t for performance! If it makes it faster/better, then bonus!
                     Capacity Planning for LAMP
                                  Architectures

For graphs you don’t have raw data for

GraphClick
http://www.arizona-software.ch/applications/graphclick/en/

   - graph digitizer package
   - $8 US
   - you pick points on a calibrated image, it spits out tabular data
Capacity Planning for LAMP
             Architectures
                      Capacity Planning for LAMP
                                   Architectures

Once you have:

1.   Time history of metrics
2.   “Ideal” peak levels (ceiling)

Then you can:

3.   Predict the future!*
Capacity Planning for LAMP
             Architectures

   Example:

Photo Processing
Capacity Planning for LAMP
             Architectures

           Photos uploaded/processed/min




           Avg processing time per minut




           Avg CPU per minute
Capacity Planning for LAMP
             Architectures
                 Capacity Planning for LAMP
                              Architectures

                        Dirty linear math

25% CPU @40 photos/min
40% CPU @60 photos/min



So….take a “ceiling”:

75% CPU @112 photos/min = 6720 photos/hour
(but double-check the process time)
Capacity Planning for LAMP
             Architectures




   Conclusion
                 Capacity Planning for LAMP
                              Architectures



   Know your machines and their limits
   Measure how the site is being used with application-
    level stats
   Tie real-world observations to server stats
                 Capacity Planning for LAMP
                              Architectures

                 Some Flickr statistics

   300M photos, 4 or 5 different sizes
   Keep ~25M images in cache at any time, ~1M from RAM
   2B MySQL queries/day
   21k req/sec to memcached
   1.2 PT raw disk storage

								
To top