Taming Deployment with SmartFrog

Document Sample
Taming Deployment with SmartFrog Powered By Docstoc
					Farms, Fabrics and Clouds

  Steve Loughran
  Julio Guijarro
  HP Laboratories, Bristol, UK

  November 2008
Steve Loughran


Researcher at HP Laboratories
Area of interest: Deployment
Author of Ant in Action




                            Page 2   www.smartfrog.org
Julio Guijarro



                                         Researcher at HP
                                          Laboratories

                                         Area of interest:
                                          Deployment

                                         In charge of OSS release

                                         http://smartfrog.org/


    Farms, Fabrics and Clouds   Page 3                       www.smartfrog.org
    Our research - see smartfrog.org

• How to host big applications
  across distributed resources
  – Automatically
  – Repeatably
  – Dynamically
  – Correctly
  – Securely
• How to manage them from
  installation to removal
• How to make dynamically
  allocated servers useful


                             Page 4    www.smartfrog.org
Question




             Who had breakfast this morning?




   Farms, Fabrics and Clouds   Page 5      www.smartfrog.org
Question




                  Who harvested wheat or corn,
                       or killed an animal
                                for
                        that breakfast?



   Farms, Fabrics and Clouds   Page 6            www.smartfrog.org
                            Farms provide food.

                It is somebody else's problem




Farms, Fabrics and Clouds           Page 7        www.smartfrog.org
Question




 Who is wearing clothes they wove or knitted
                themselves?




   Farms, Fabrics and Clouds   Page 8   www.smartfrog.org
Provisioning of clothing -fabrics- is outsourced

                   It is somebody else's problem




   Farms, Fabrics and Clouds    Page 9             www.smartfrog.org
Future applications are on the Web


                 • Web Browser, AJAX clients
                 • Richer: Flash, XUL, Silverlight
                 • "… as a Service "


                 ⇒Lots of code running in the server
                 ⇒Unpredictable demand
                 ⇒Data mining/analysis problems
   Farms, Fabrics and Clouds     Page 10               www.smartfrog.org
Old world installation: single server
                      Application server
                      Database
                                                         Single web server,
                                                         Single DB
 firewall/
  router
                                                         RAID filestore



                                                         -SPOF
                                                         -limitations of scale
 firewall/
  router                      Application   Database
                              server        server



        Farms, Fabrics and Clouds              Page 11                  www.smartfrog.org
yesterday: clustering
                                   firewall/
                                    router
                                                              Multiple web servers,
                                               Application
                                                              Replicated DB
                                               servers
                                                              RAID Network filestore
                                                              Load-balancing router

                                                              -Cost
                     replication                              -Complexity
                     protocol
                                                              -Limitations of scale
 DB Master                         DB Slave

              Maintains the illusion of a single server
     Farms, Fabrics and Clouds                      Page 12                   www.smartfrog.org
Now: server farms



500 web servers,
Distributed filestore
Rented storage & CPU

Scales up
No capital outlay
Agile infrastructure


    Farms, Fabrics and Clouds   Page 13   www.smartfrog.org
Tomorrow? grid fabric. 50000 servers




   Farms, Fabrics and Clouds   Page 14   www.smartfrog.org
Application architectures and deployment
 problems change radically in this world




 Farms, Fabrics and Clouds   Page 15   www.smartfrog.org
                                                                        REST APIs
Application architectures
  Chrome
  Your friends
  are having
                     Mozilla
                     Your friends
                     are having
                                    IE

                                    Your friends are
                                                       iPhone

                                                       You are having
                                                       more fun than
                                    having more fun
  more fun           more fun                          your
                                    than you           friends
  than you           than you
                                                                                         Affiliate View

                                                                                         David Hasslehof
                                                                        Diskless front   keywords cost
                                                                             end         more today

                                                                         Memcached       Management view
                                                                           JSP?
                  Scatter/gather?                                          PHP?          David Hasslehof is
                 Message Queue?                                                          very popular today
                  Tuple Space?

                                                                                         Developer View

                                                                                         MR job 17 completed.




                                                                         Cloud Layer     Ops View
                                                                                         92% servers up
                                                                           HDFS          Network OK
                                                                           HBase         Cost: $200/hour
                                                                         MapReduce
                                                                          DLucene
                                                                                         Cloud Owner View
                                                                                         Customer #17 is
                                                                                         using lots of disk
                                                                                         space. Normal
                                                                                         HDD failure rate

    September 2008
Application architectures



          •   ROA/REST
          •   Virtualized
          •   MapReduce
          •   Shards
          •   Tuple-spaces
          •   XMPP



    Farms, Fabrics and Clouds   Page 17   www.smartfrog.org
Virtualization




    Farms, Fabrics and Clouds   Page 18   www.smartfrog.org
Why?


•   Save on hardware (and power, space)
•   Dynamically move running servers
•   Demand creation of new images
•   Testing complex system configurations
•   Redistributing entire machine image
•   'virtual appliance'




       Farms, Fabrics and Clouds   Page 19   www.smartfrog.org
Assumptions that are now invalid


•   Systems have a long lifespan
•   It is slow/expensive to create a new system
•   It is expensive to duplicate one
•   Systems can/should be managed by hand
•   Clocks proceed at the same rate
•   Physical RAM doesn’t get swapped out
•   Running machines can't be moved/cloned
•   Virtualization is only for testing.


       Farms, Fabrics and Clouds   Page 20        www.smartfrog.org
Server Farms




                                         http://www.linuxjournal.com/
   Farms, Fabrics and Clouds   Page 21                      www.smartfrog.org
Assumptions that are now invalid



•   System failure is an unusual event
•   100% availability can be achieved
•   Data is always near the server
•   You need physical access to the servers
•   Databases are the best form of storage
•   You need millions of $/£/€ to play



       Farms, Fabrics and Clouds   Page 22    www.smartfrog.org
Who has the servers?



•   Yahoo!, Google, MSN, Amazon, eBay: services
•   MMORPG Game Vendors:
      World of Warcraft, Second Life
•   EU Grid: Scientists
•   HP, IBM, Sun: rent to companies (some resold)
      -focus on CPU performance for enterprise
•   Amazon: rent to anyone with an Amazon account
      -focus on startups

      Farms, Fabrics and Clouds   Page 23   www.smartfrog.org
Amazon EC2


•   Pay as you go Virtual Machine Hosting
•   No persistent storage other than S3 filestore -
    uses HTTP GET/PUT/DELETE operations
•   $0.10 per CPU/hour
•   Resold OS images for more
     (RedHat, Windows)
•   Rent static IP addresses for failover/balancing
•   New: RAID-like storage


       Farms, Fabrics and Clouds   Page 24      www.smartfrog.org
Amazon EC2

                                               Public Internet

                       $                  $                          $              $
                                  Host                                      Host

                  AMI                    AMI                       AMI             AMI
            (Xen VM)                 (Xen VM)                    (Xen VM)      (Xen VM)                       $

                       /mnt               /mnt                       /mnt           /mnt


                                         Fast (free) network



 free access; slow initial read time                                                     pay per GET; per
                                                                                                megabyte
                                                   S3 Storage

      Farms, Fabrics and Clouds                        Page 25                            www.smartfrog.org
Demo




  Farms, Fabrics and Clouds   Page 26   www.smartfrog.org
EC2 Limitations



•   Can't talk to peers using public IP addresses
•   Persistent file system is a premium extra
•   Most addresses are dynamic
•   No managed redundancy/restart
•   No multicast IP
•   No movement of VMs off high-traffic racks



       Farms, Fabrics and Clouds   Page 27      www.smartfrog.org
Amazon S3



•   Multiple geo-located data storage
•   No limits on size
•   Cost of write is high (guarantee of written remotely)
•   Read is cheap; may be out of date
•   Cost: Low

S3 is a global file system that any project can afford


       Farms, Fabrics and Clouds   Page 28       www.smartfrog.org
Farms, Fabrics and Clouds   Page 29   www.smartfrog.org
MapReduce




 Commodity data processing for commodity data
  Farms, Fabrics and Clouds   Page 30   www.smartfrog.org
Assumptions that are now invalid



•   Terabyte datasets are hard to work with
•   Code runs on a single machine
•   Sequential code is better than parallel code
•   RAID hardware is the best way to store data
•   Databases are better than filesystems
•   Low-value data isn't worth collecting
    even if you don't have a use for it now



       Farms, Fabrics and Clouds   Page 31     www.smartfrog.org
Shards




   Farms, Fabrics and Clouds   Page 32   www.smartfrog.org
Assumptions that are now invalid

• A single farm needs to scale to infinity
• You need to provide 100% availability to 100%
  of users
• You have to roll out simultaneous updates to
  the application, changes to the DB schema,
  globally




    Farms, Fabrics and Clouds   Page 33     www.smartfrog.org
XMPP




                              post extends GoogleChatClientWorkflow {
                                            to "smartfrog.two@gmail.com";
                                            login "smartfrog.two";
                                            password xmpp.password;
                                            message "hello, world";
                              }



  Farms, Fabrics and Clouds       Page 34                             www.smartfrog.org
Assumptions that are now invalid

•   You can't send message to a laptop that moves
    around behind a firewall.
•   You need to build your own monitoring
    infrastructure.
•   Blocking RPC is a good metaphor for long-haul
    communications.
•   You can't send messages to your server farm from
    your phone
•   IT doesn't have their eyes on your protocol


      Farms, Fabrics and Clouds   Page 35    www.smartfrog.org
Problems for us farmers

•   Power management
•   Predictive disk failure management
•   Load balancing for availability, power
•   File management
•   Billing
•   Routing
•   Security/Isolation
•   How will this change server hardware?
•   Managing/Configuring Machine Images
•   Diagnostics when things go wrong
      Farms, Fabrics and Clouds   Page 36    www.smartfrog.org
“Agile” Routers

•   Handle hundreds to thousands of (concurrent)
    change requests/second
•   Integrate with billing
•   Managed throttling to specific hosts
•   Propagation of state to peer rooters
•   'agile' DHCP -short leases; mobile
•   Monitored bandwidth may trigger VM migration




      Farms, Fabrics and Clouds   Page 37    www.smartfrog.org
“Agile” Operating Systems

•   Design for VM-only use
•   Limited functionality
•   Limited lifespan
•   Fully configurable before initial boot
•   Adapt to changes in surrounding environment
•   Viable licensing model




      Farms, Fabrics and Clouds   Page 38    www.smartfrog.org

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:3
posted:2/15/2012
language:
pages:38