Distributed Computing Paradigms by wuzhenguang


									Distributed Computing
               Lecture 2

   Characteristics of Distributed Applications
       Message Passing Paradigm
       Client/Server Paradigm
       Peer-to-Peer Paradigm
   Remote Procedure Call
   Distributed Object Paradigm
       RMI
       Network Service
       Object Request Broker
   High Performance Computer
       Cluster Computer
       Cluster Architecture
       Cluster Components
       Cluster Software
    2                                        Distributed Computing Paradigm
Characteristics of Distributed Applications

   Characteristics that distinguish distributed
    applications from conventional applications which run
    on a single machine.
       Interprocess communication: A distributed application
        require the participation of two or more independent entities
        (processes). To do so, the processes must have the ability to
        exchange data among themselves.
       Event synchronization: In a distributed application, the
        sending and receiving of data among the participants of a
        distributed application must be synchronized.

    3                                            Distributed Computing Paradigm
 Distributed Application Paradigms
level of abstraction

                                           object space
                       network services, object request broker, mobile agent
                         remote procedure call, remote method invocation

                                         message passing


   4                                                       Distributed Computing Paradigm
The Message Passing Paradigm
   Message passing is the most fundamental paradigm for
    distributed applications.
       A process sends a message representing a request.
       The message is delivered to a receiver, which processes the
        request, and sends a message in response.
       In turn, the reply may trigger a further request, which leads
        to a subsequent reply, and so forth.

    5                                            Distributed Computing Paradigm
The Message Passing Paradigm

Process A
                              Process B

                                                          a message

            Message passing

 6                                        Distributed Computing Paradigm
The Message Passing Paradigm
   The basic operations required to support the basic
    message passing paradigm are send and receive.
       For connection-oriented communication, the operations
        connect and disconnect are also required.
       The interconnected processes perform input and output to
        each other, in a manner similar to file I/O.
   The socket application programming interface is based
    on this paradigm.

    7                                           Distributed Computing Paradigm
The Client-Server Paradigm
   Perhaps the best known paradigm for network
    applications, the client-server model assigns
    asymmetric roles to two collaborating processes.
       One process, the server, plays the role of a service provider
        which waits passively for the arrival of requests.
       The other, the client, issues specific requests to the server and
        awaits its response.

    8                                               Distributed Computing Paradigm
Role of Server
   Hold information
   Provide services
   Starts before client
   Wait for client request
   Carry out request
   Return result (reply) to client
   Never make use of clients
   Can make requests to other servers (delegation)

    9                                   Distributed Computing Paradigm
Role of Client
   Starts communication
   Must have server destination ready (server starts up
   Makes requests to servers
   Never use other clients
   Do not carry out requests, only issues them

    10                                   Distributed Computing Paradigm
The Client-Server Paradigm
                                service request
                                a client process
                                 a server process
     Server host                 a service

                                                    Client host

              The Client-Server Paradigm, conceptual

11                                                  Distributed Computing Paradigm
The Client-Server Paradigm
   By assigning asymmetric roles to the two sides, event
    synchronization is simplified:
        the server process waits for requests
        the client in turn waits for responses.
   Many Internet services are client-server applications.
        These services are often known by the protocol that the
         application implements.
        Well known Internet services include HTTP, FTP, DNS,
         finger, gopher, etc.

    12                                             Distributed Computing Paradigm
Thin Client vs. Fat Client
   Partitioning of software between client and server
   Thin Client
        client processing is small, bulk takes place at server
        client hardware requirements are less
   Thick (or Fat) Client
        client performs bulk of processing with server providing basic
         tasks, e.g. disk storage and retrieval
        e.g. PC connection to internet.
            This architecture provides a practical solution but creates an ever
             increasing need for faster processors, high capacity storage and
             complex operating systems
   Trade off between the two ?

    13                                                    Distributed Computing Paradigm
Peer-to-Peer System Architecture
   Peer-to-peer is an architecture where computer
    resources and services are directly exchanged between
    computer systems.
   These resources and services include the exchange of
    information, processing cycles, cache storage, and disk
    storage for files.
   In such an architecture, computers that have
    traditionally been used solely as clients communicate
    directly among themselves and can act as both clients
    and servers, assuming whatever role is most efficient
    for the network.

    14                                   Distributed Computing Paradigm
Peer-to-Peer Model

15                   Distributed Computing Paradigm
Peer-to-Peer Distributed Computing Paradigm

   In the peer-to-peer paradigm,               process 1
    the participating processes
    play equal roles, with
    equivalent capabilities and
    responsibilities (hence the     re que st     re que st

    term “peer”).                           re sponse
                                                              re sponse
   Each participant may issue a
    request to another
    participant and receive a
    response.                                   process 2

    16                                          Distributed Computing Paradigm
Peer-to-Peer Applications
   Whereas the client-server paradigm is an ideal model
    for a centralized network service, the peer-to-peer
    paradigm is more appropriate for applications such as
        instant messaging
        peer-to-peer file transfers
        video conferencing
        collaborative work.
   It is also possible for an application to be based on
    both the client-server model and the peer-to-peer

    17                                     Distributed Computing Paradigm
Peer to Peer examples
   Napster.com – an example of peer-to-peer file transfer
        allow files (primarily audio files) to be transmitted among
         computers on the Internet.
        It makes use of a server for directory in addition to the peer-
         to-peer computing.
   Bit Torrent Protocol
   For developing web applications of instant message
    and resource sharing, there are open protocols and
    tools available on the web.
         JXTA (jxta.org)
        Jabber (jabber.org)

    18                                               Distributed Computing Paradigm
Distributed Programming Paradigm
   Remote Procedure Call (RPC)
   Distributed Object Paradigm
        RMI
        Network Service
   Object Request Broker
        CORBA
        DCOM
        Java Bean

    19                            Distributed Computing Paradigm
Remote Procedure Call
   The Remote Procedure Call (RPC) allows distributed software to be
    programmed in a manner similar to conventional applications
    which run on a single processor.
   Using this model, interprocess communications proceed as
    procedure, or function, calls, which are familiar to application
   A remote procedure call involves two independent processes, which
    may reside on separate machines.
     A process, A, wishing to make a request to another process, B, issues
      a procedure call to B, passing with the call a list of argument values.
     At the completion of the procedure, process B returns a value to
      process A.

    20                                                Distributed Computing Paradigm
Remote Procedure Call
                                                   Process B
Process A

              proc1(arg1, arg2)



 21                                 Distributed Computing Paradigm
Tools for RPC
   There are two prevalent APIs for Remote Procedure
        The Open Network Computing Remote Procedure Call,
         evolved from the RPC API originated from Sun Microsystems in
         the early 1980s.
        The Open Group Distributed Computing Environment (DCE)
   Both APIs provide a tool, rpcgen, for transforming
    remote procedure calls to local procedure calls.

    22                                           Distributed Computing Paradigm
The Distributed Objects Paradigms
   The idea of applying object orientation to distributed
    applications is a natural extension of object-oriented
    software development.
   Applications access objects distributed over a network.
   Objects provide methods, through the invocation of
    which an application obtains access to services.
   Object-oriented paradigms include:
        Remote method invocation (RMI)
        Network services
        Object request broker

    23                                    Distributed Computing Paradigm
Remote Method Invocation (RMI)
   Remote method invocation is the object-oriented
    equivalent of remote procedure calls.
   In this model, a process invokes the methods in an
    object, which may reside in a remote host.
   As with RPC, arguments may be passed with the

    24                                   Distributed Computing Paradigm
Remote Method Invocation (RMI)
                                       Process 2
     Process 1
            remote method invocation      method1

                                       a remote object

         The Remote Method Call Paradigm

25                                      Distributed Computing Paradigm
The Network Services Paradigm
   In this paradigm, service providers register themselves with
    directory servers on a network.
   A process desiring a particular service contacts the directory server
    at run time, and, if the service is available, will be provided a
    reference to the service. Using the reference, the process interacts
    with the service.
   This paradigm is essentially an extension of the remote method call
     The difference is that service objects are registered with a global
       directory service, allowing them to be look up and accessed by
       service requestors.
   Java’s Jini technology is based on this paradigm.
     http://www.sun.com/software/jini/

    26                                             Distributed Computing Paradigm
The Network Services Paradigm
                     Directory service

                                                   service object

     Service requestor

27                                       Distributed Computing Paradigm
The Object Request Broker Paradigm
   In the object broker paradigm , an application issues
    requests to an object request broker (ORB), which directs
    the request to an appropriate object that provides the
    desired service.
   The paradigm closely resembles the remote method
    invocation model in its support for remote object access.
     The difference is that the object request broker in this
      paradigm functions as a middleware which allows an
      application, as an object requestor, to potentially access
      multiple remote (or local) objects.
   The request broker may also function as an mediator for
    heterogeneous objects, allowing interactions among objects
    implemented using different APIs and /or running on
    different platforms.
    28                                       Distributed Computing Paradigm
Common Object Request Broker Architecture


                            Object Request Broker

   This paradigm is the basis of the Object Management Group’s
    CORBA (Common Object Request Broker Architecture)
        http://www.corba.org/
   CORBA in Java
        http://java.sun.com/developer/onlineTraining/corba/
   Java IDL: http://java.sun.com/products/jdk/idl/index.jsp
    29                                              Distributed Computing Paradigm
Other Object-Based
(or Component-Based) Paradigm
   Component-based technologies such as Microsoft’s
    COM, Microsoft DCOM, Java Bean, and Enterprise Java
    Bean are also based on distributed-object paradigms
        Components are essentially specialized, packaged objects
         designed to interact with each other through standardized
   In addition, application servers, popular for
    enterprise applications, are middleware facilities
    which provide access to objects or components.

    30                                            Distributed Computing Paradigm
High Performance

Rise and Fall of Computer Architectures
   Vector Computers (VC) - proprietary system:
      provided the breakthrough needed for the emergence of
        computational science, buy they were only a partial answer.
   Massively Parallel Processors (MPP) -proprietary systems:
      high cost and a low performance/price ratio.
   Symmetric Multiprocessors (SMP):
      suffers from scalability
   Distributed Systems:
      difficult to use and hard to extract parallel performance.
   Clusters - gaining popularity:
      High Performance Computing - Commodity Supercomputing
      High Availability Computing - Mission Critical Applications

    32                                             Distributed Computing Paradigm
Architecture Share in Top500.org

33                       Distributed Computing Paradigm
  Computer Food Chain: Causing the demise
  of specialize systems

•Reduction in numbers of mainframes, supercomputers, & MPPs
    34                                 Distributed Computing Paradigm
Why PC/WS Clustering Now ?
   Individual PCs/workstations are becoming increasing powerful
   Commodity networks bandwidth is increasing and latency is
   PC/Workstation clusters are easier to integrate into existing
   Typical low user utilization of PCs/WSs
   Development tools for PCs/WS are more mature
   PC/WS clusters are a cheap and readily available
   Clusters can be easily grown

    35                                           Distributed Computing Paradigm
Architecture & Technologies
   Leveraging of workstation technologies
        Processors
            Intel Pentium Processors
        Interconnections networks
            Fast Ethernet
            Gigabit Ethernet
            Myrinet
        Operating systems
        Programming languages
        Compilers

    36                                  Distributed Computing Paradigm
Clusters: What are they?

    A collection of computers connected over a local
     network and appear as a single computer system
    An approach to build high-performance computers at
     relatively low cost by using commodity components

37                                        Distributed Computing Paradigm
What is Cluster ?
   A cluster is a type of parallel or distributed processing
    system, which consists of a collection of interconnected
    stand-alone computers cooperatively working together as a
    single, integrated computing resource.
   A node is a single or multiprocessor system with memory,
    I/O facilities, & OS
   A cluster
        generally 2 or more computers (nodes) connected together
        in a single cabinet, or physically separated & connected via a LAN
        appear as a single system to users and applications
        provide a cost-effective way to gain features and benefits

    38                                                 Distributed Computing Paradigm
Beowulf Clusters

What is a Beowulf ?
   Massively parallel computer built out of COTS
    (Commercial Of The Shelf)
   Runs a free operating system
   Connected by high speed interconnect
   Compute nodes are dedicated
   Cheaper cluster because everything in a Beowulf is
    open-source and open standard

    40                                   Distributed Computing Paradigm
Beowulf: The Beginning
   Thomas Sterling and Donald Becker (Center of Excellence in
    Space Data and Information Sciences, NASA Goddard Space Flight
   Summer 1994: built an experimental cluster
   Called their cluster Beowulf

    41                                            Distributed Computing Paradigm
The First Beowulf
   16 x 486DX4, 100MHz
   16MB of RAM each,
    256MB in total
   Channel bonded
    Ethernet (2 x 10Mbps)

    42                      Distributed Computing Paradigm
Current Beowulfs

 43                Distributed Computing Paradigm
Cluster: Rack Mount Server

      front     back               room
 44                      Distributed Computing Paradigm
Meteor Cluster at SDSC
                            Rocks v2.2
                            2 Frontends
                            4 NFS Servers
                            100 nodes
                                  Compaq
                                        800, 933, IA-64
                                        SCSI, IDA
                                  IBM
                                        733, 1000
                                        SCSI
                            50 GB RAM
                            Ethernet
                                  For management
                            Myrinet 2000

45                           Distributed Computing Paradigm
Basic Cluster System Architecture

46                     Distributed Computing Paradigm
Cluster Architecture
                                                         Parallel Applications
                                                            Parallel Applications
                                                            Parallel Applications
  Sequential Applications
     Sequential Applications
      Sequential Applications                      Parallel Programming Environment

                                       Cluster Middleware
                  (Single System Image and Availability Infrastructure)

 PC/Workstation             PC/Workstation          PC/Workstation                PC/Workstation

      Communications           Communications          Communications                 Communications
         Software                 Software                Software                       Software

      Network Interface        Network Interface       Network Interface              Network Interface
         Hardware                 Hardware                Hardware                       Hardware

                          Cluster Interconnection Network/Switch

 47                                                                        Distributed Computing Paradigm
Cluster Interconnect
   The most important component
   Factors to consider
        Bandwidth
        Latency
        Price
        Software support

    48                             Distributed Computing Paradigm
Cluster Software

Example: Roadrunner System Software
   Redhat Linux 5.2 (6.0)
   SMP Linux kernel 2.2.10
   MPI (Argonne’s MPICH
   Portland Group Compiler Suite
   Myricom GM Drivers (1.04) (Intrconnect-Myrinet)
   Portable Batch Scheduler (PBS)

    50                                 Distributed Computing Paradigm
Cluster Software
   We’ll concentrate on the cluster management core
   Three choices:
        Vanilla Linux/FreeBSD
        Free cluster management software (a very patched up
        Commercial cluster management software (a very patched
         up Linux)

    51                                      Distributed Computing Paradigm
The issues
   Beowulfs can get very large (100’s of nodes)
   Compute nodes should setup themselves automatically
   Software updates must be automated across all the
   Software coherency is an issue

    52                                 Distributed Computing Paradigm
Free cluster management softwares
   Rocks: http://rocks.npaci.edu (Rock and Roll)
   Oscar: http://oscar.sourceforge.net/
   MOSIX: http://www.mosix.org/
   Linux that comes with software for cluster
    management (usually patched)
   Reduces dramatically the time needed to get things up
    and running
   Open source, but if something breaks, you have one
    more piece of software to hack

    53                                  Distributed Computing Paradigm
Rocks Cluster Software Stack

54                       Distributed Computing Paradigm
Common to Any Cluster

55                      Distributed Computing Paradigm
System Software
   Operating Systems
   Compilers
   Parallel Programming Environment
   Job Scheduling

    56                                 Distributed Computing Paradigm
OS : Red Hat Linux
   Stock Red Hat 7.2 with updates (x86 and ia64)
   Linux 2.4 Kernel
   No support for other distributions
        We believe Red Hat is the market leader for Linux
             In the US
             And becoming so in Europe
        Automated install (kickstart) is a requirement
        Very good hardware detection

    57                                           Distributed Computing Paradigm
MPI (Message Passing Interface)

   Old Standard (1.1, June 1995)
   Portable, practical
   Freely-available reference implementations
   Version 2.0 includes parallel I/O, one-sided
    communication, etc.

    58                                    Distributed Computing Paradigm
   NFS is a software that supports sharing of files over a
   A network file system should appear to its users to be a
    conventional, centralized file system.
   NAS (Network-attached storage) and SAN (Storage
    Area Network)
   NIS consists of a client-server directory service
    protocol for distributing system configuration data
   NIS manage user accounts and host information (host)
    names between computers on a network.
   A network information service has been mostly
    replaced by a more modern and secured system, LDAP

    59                                    Distributed Computing Paradigm
Open SSH
   Replaces Telnet, Rsh
        Cryptographically strong authentication and encryption
        Forwards X11 connections (no more $DISPLAY)
   Rocks uses SSH
        Mpi-launch
        Cluster-fork
   Ssh-agent
        Manager for SSH keys
        ssh-agent $SHELL

    60                                       Distributed Computing Paradigm
   DHCP = Dynamic Host Configuration Protocol
   DHCP is a network application protocol used by
    devices (DHCP clients) to obtain configuration
    information for operation in an Internet Protocol
   DHCP reduces system administration workload,
    allowing devices to be added to the network with
    little or no manual intervention.
   DHCP
        Manages network information
        Alternative to static IP address
        Foundation of insert-ethers

    61                                      Distributed Computing Paradigm
Real issue 1 - space
   Getting a Beowulf is great, but do you have the space to
    put it?
   Often space is at a premium, and Beowulf is not as
    dense as traditional supercomputers
   Rackmount? Extra cost! e.g. cabinet ~$1500, case for
    one node ~$400

    62                                    Distributed Computing Paradigm
Real issue 2 – heat management
   The nodes, with all the high powered processors and
    network cards, run hot
   Especially true for Athlons - can reach 60°C
   If not properly managed, the heat can cause crash or
    even hardware damage!
   Heatsink/fans – remember to put in in the right

    63                                   Distributed Computing Paradigm
Real issue 3 - power
   Do you have enough power in your room?
   UPS? Surge protection?
   You don’t want a thunderstorm to fry your Beowulf!

    64                                  Distributed Computing Paradigm
Real issue 4 - noise
   Beowulfs are loud. Really loud.
   You don’t want it on your desktop.

                                          Bad idea

    65                                   Distributed Computing Paradigm
Real issue 5 - cables
   Make the cables discipline.
   Color scheme your cables!

    66                            Distributed Computing Paradigm
High End Clusters

     Top 5
Top500.org (June 2010)
1.   Jaguar - 1.75 Petaflops, USA
2.   Nebulae – 1.27 Petaflops, China
3.   Roadrunner – 1.04 Petaflops, USA
4.   Kraken - 831 Teraflops, USA
     Jugene (BlueGene/P) - 825 Teraflops,

               2                    3        4                           5
     68                                     Distributed Computing Paradigm
How supercomputers are connected ?
     How supercomputer are built ?
                           Node Card
                         32 chips, 4x4x2
                    16 compute, 0-2 I/O cards
                          90/180 GF/s
     Compute Card            16 GB
     2 chips, 1x2x1
     5.6/11.2 GF/s
         1.0 GB

Chip (node)
2 processors
2.8/5.6 GF/s
                                                     Rack          64 Racks,
                                                32 Node cards     64x32x32
                                                 2.8/5.6 TF/s    180/360 TF/s
                                                   512 GB            32 TB

To top