Capn,-Im-givin-er-all-shes-got! by akgame


More Info
									"Cap'n, I'm givin' 'er all she's
  What to do when your servers are at
              their limit.
• How many times did
  Scotty have to be a
• He just needed bigger,
  more efficient
         Web server capacity
• Your web server is an
• With heavy usage,
  performance degrades
• The server can stop
  responding altogether
Our Story
     • Here’s how we
       reached capacity…
     • …and re-designed our
       system to make sure it
       doesn’t happen again.
             Over Capacity
• Site access growing at 25 - 50% per
• First week of school was the biggest
  problem, especially opening day.
• More key resources on-line.
• Poor diagnostics
• “Seemed slower”
First steps--bulk up
       Bulking up the server
• Add disk--”Disk is cheap.”
• Add memory--”Memory is cheap.”
• Get a bigger machine!
              Short-term fix
• Got us through a few
  semesters, but we hit
  the new limit.
• Next size machine was               Quic kT ime™ and a
                          T IF F (Unc om pres sed) dec ompres sor

  very expensive
                              are needed to see t his pict ure.

• Bigger hardware
  wasn’t working
     Where are the resources?
• Perl CGI scripts; many on-line resources
  that were developed over time.
• CGI was easy to write, but very resource
• Then we added database interaction,
  opening and closing DB handles each time.
               Each CGI hit
• spawn external
• invoke perl
• possibly make DB
• return content to web
• clean up
Where can we improve?
           • Look for ways to
             speed up CGI:
             – already many scripts.
           • Wanted to leverage
             existing code.
          mod_perl Solution
• In our case, mod_perl was a very good
• Able to use most of our existing code.
• Works with apache, and that was the server
  platform we were running.
           mod_perl Solution
• Apache::Registry to speed up the CGI scripts;
• Apache::DBI to cache DB connections.
             Getting Started
• But it’s complicated--
  how do we get started?
        Call the Consultants
• Called Perl trainers to give us mod_perl
• Helped us with training and code review.
• Worked with us on server configuration.
• Confirmed for management that the
  technology was accepted and widely used.
  Analysis: What a server does.
• Receives request from browser;
• Invokes some resource to generate content
  to respond;
• Some responses are fast, others are slow;
• Some are dynamic, others are static;
        Where are you slow?
• We looked at what parts of our response
  cycle were slow;
• 80/20 rule says you 80% of your
  performance problem is in 20% of your
• Find those parts.
• Make sure you solve the right problem.
            mod_perl speed
• For us, mod_perl accelerated the slow parts.
• mod_perl handles:
  – persistent perl interpreter;
  – keeps code in memory;
  – Apache::DBI keeps persistent DB connections.
• Slowest parts are all very fast now.
    Where are you slow, part II
• Needed to convert our CGI scripts to run
  under mod_perl.
• Again, 80/20. Generated a top 10 list from
  our server logs.
• Put effort into converting those 10
              Technical Parts
• End of manager
  portion of presentation
All that memory?
        • mod_perl loads a
          bunch of code into
        • Apache processes are
        • Running out of
        Server modifications
• Server does different things: application
  processing and serving flat files (images
  and HTML)
• Don’t need a huge application process to
  serve an image or HTML page.
      Solution: split the server
• Set up two apache servers:
• one proxy server on the front
• one application server on the back
              Front server
• Runs on normal ports (80 and 443)
• Responds to all requests
• Caches images and serves them if it can
• Sends app requests through to back
• Small footprint: very few modules and no
• Many children available
               Back server
• Application server running on alternate port
• Only responds to front server
• Large processes with mod_perl and all
  loaded code
• Small number of these processes
            SSL Resources
• SSL is CPU intensive (key generation for
  encryption between browser and server)
• Can go with faster CPUs
• Offload processing to dedicated hardware
  component (cryptographic accelerator
            We’re set, right?
• Web server is wicked
• Hardly working even
  during heavy load;
• We’re done…aren’t
  What if the server goes down?
• Enterprise had two engines;
• If one goes down, there is a backup;
• We needed the same thing so a machine
  failure wouldn’t interrupt service.
     • At least two
       components for each
       one we had;
     • So the web server
       machine had to
       become at least two
      Cold Backup Machine?
• Could buy a second machine and turn it on
  in case of failure (cold backup).
• Server was big and expensive, so a backup
  would also be expensive.
• The backup would just sit there.
             Server Farming?
• Created a series of
  parallel servers all
  running at the same
• Often called a server
Basic Farm Layout
            Extra Hardware
• Some sort of load balancing:
  – Can be software running on a box
  – We use dedicated hardware
• Multiple boxes:
  – Instead of one big server, many small ones
  – Each is cheaper
  – Impact of losing one is smaller
            Code Challenges
• Apps must act the same if user is routed to a
  different server:
  – Can’t rely on flat files, unless you have shared
    file system.
  – Could use “sticky” settings to keep users on the
    same machine, but this can get messy.
• Same code to all
• Same server config on
  all machines
• Automated our
  deployment process
  and combined with
  version control
         Where we are today
• 3 Cisco load balancers
• 4 Sun web farm machines:
  – mod_perl;
  – some regular cgi;
  – html and images;
• Oracle database in cluster (2 machines)
• Automated code deployment system
Extra Content
       • Additional
            SSL negotiation
• Probably want to do SSL on front server:
  – SSL is CPU intensive;
  – If you run servers on different machines, you
    don’t want to contend with app server.
           SSL negotiation
• Don’t encrypt traffic from front to back
  server--unnecessary performance hit.
• If you are running servers on the same
  machine, not a problem.
• If on different machines, network must be
               IP Stickiness
• If your load balancer has IP sticky, use it.
• Prevents renegotiation for the same user
  getting moved around.
• Can be huge hit.
• Use keep-alive also for images.
            Server Tuning
• Watch your server to determine how to fine-
• Apache very flexible, but needs to be
  configured properly.
• Continue to look for that big performance
• Keep metrics to identify if something
           Shared Database
• May need database so share data across all
• Critical dependency.
• Need redundancy/high availability at this
  level too.
           High Availability
• Storage Area Network (SAN)
• Special device with:
  – Guaranteed high availability;
  – Fiber networking for very fast access.
  – Large storage.
            Server Monitoring
•   Apache has modules for monitoring;
•   Very useful and worth the performance hit;
•   Apache::VMonitor
•   server-info, server-status, perl-status

To top