Docstoc

Decentralized Resource Availability Prediction in Peer-to-Peer

Document Sample
Decentralized Resource Availability Prediction in Peer-to-Peer Powered By Docstoc
					Decentralized Resource Availability Prediction in
          Peer-to-Peer Desktop Grids



                 Karthick Ramachandran

      Supervisors: Dr. Hanan Lutfiyya and Dr. Mark Perry
   Computer Science Department, University of Western Ontario




                        Thesis Defense
                        May 01, 2009
Motivation       Focus    Architecture        Implementation          Verification and Validation          Conclusion


                                             Outline

      1      Motivation

      2      Focus

      3      Architecture

      4      Implementation

      5      Verification and Validation

      6      Conclusion



                                  Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                                 2 / 43
Motivation   Focus   Architecture        Implementation          Verification and Validation          Conclusion


                                    Motivation




      “Companies across US are wasting $2.8 billion every year powering
                1.8 million unused computers during night”
                             - PC Energy report 2009

                             Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                            3 / 43
Motivation      Focus    Architecture        Implementation          Verification and Validation          Conclusion


                            Volunteer Computing


     Volunteer computing (desktop grid) is a type of distributed
     computing in which computer owners donate their computing
     resources (such as processing power and storage) to one or more
     projects.


     Volunteer Computing Characteristics
             Leverages the idle CPU cycles for number crunching tasks
             Answer to the demand for huge processing power




                                 Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                                4 / 43
Motivation      Focus    Architecture        Implementation          Verification and Validation          Conclusion


                                        The Problem




                                                   ?
             Can the system behave the same way irrespective of the number of
             participating nodes?
             What is the guarantee that the peer selected for job execution will
             not terminate the execution because of some user activity?




                                 Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                                5 / 43
Motivation      Focus    Architecture        Implementation          Verification and Validation          Conclusion


                                        Related Work

             BOINC (Anderson et.al) - Client/Server with no resource selection
             policy
             Xtremweb (Fedak et.al)- Client/Server with no resource selection
             policy
             Kim et.al - Peer to Peer desktop grid with no resources selection
             policy
             Brevik et.al - Mathematical model for resource availability
             prediction with no focus on the type of architecture
             RIDGE (Budati et.al) - Client/Server with Reliability Ratings
             TMDG (L Shang et.al) - Client/Server with Trust Model to predict
             machine availability



                                 Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                                6 / 43
Motivation      Focus    Architecture        Implementation          Verification and Validation          Conclusion


                                Focus of our work


     The focus of this system is to create a desktop grid
             that minimizes centralization of resource selection process
             that can do resource availability prediction
     We build a P2P desktop grid framework which does resource
     availability prediction.
     None of the work we surveyed has taken approach of performing
     availability prediction in a P2P environment.




                                 Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                                7 / 43
Motivation       Focus    Architecture        Implementation          Verification and Validation          Conclusion




      1      Motivation

      2      Focus

      3      Architecture

      4      Implementation

      5      Verification and Validation

      6      Conclusion



                                  Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                                 8 / 43
Motivation      Focus    Architecture        Implementation          Verification and Validation          Conclusion


                                          Overview


     Components of the System
             Servers
                  Management Server
                  Job Server
                  Monitoring Server
                  Result Server
             Client - Voluntary Cloud Computing (VCC) Software
                  Individual Components
                  Job, Mobile Agent




                                 Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                                9 / 43
Motivation   Focus           Architecture            Implementation          Verification and Validation          Conclusion


                                                  Overview


                 Management Server




                                                           Peers of the System
                        Job Server




                     Monitoring Server




                       Result Server




                               Figure: Birds eye view of the system
                                         Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                                        10 / 43
Motivation      Focus    Architecture        Implementation          Verification and Validation          Conclusion


                                            Servers
     Management Server
             Management server is hosted by grid service provider
             Stores the FQDN (fully qualified domain name), organization name
             and password corresponding to each participating job server
             There is one centralized management server servicing all

     Job Server
             Job server maintains a repository of the jobs to be executed
             Organization signs up with the management server and registers its
             job server(s) with the management server
             Job server also runs the resource selection steps to find a peer node
             and submits the job to that peer node

                                 Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                                11 / 43
Motivation      Focus    Architecture        Implementation          Verification and Validation          Conclusion


                                            Servers

     Monitoring Server
             An organization can have multiple monitoring server
             Job is sent with the monitoring server and result server addresses
             During job execution, the mobile agent associated with job sends
             the status of execution

     Result Server
             After the execution of a job, the results are sent back to the result
             server
             The result of the job could be data in XML form representing the
             result of an operation



                                 Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                                12 / 43
Motivation   Focus     Architecture           Implementation             Verification and Validation      Conclusion


                            VCC Client Software
     VCC Client software is installed on each resource in the framework


                               Code Executor in Sandbox (Java Virtual Machine)




                                                     Monitoring and Prediction Engine

                     Resource Discovery
                          Module
                                                                   Resource and Group
                                                                     Availability data




                                          Peer Communication Module



                                          Underlying P2P Framework




                                 Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                                13 / 43
Motivation      Focus    Architecture        Implementation          Verification and Validation          Conclusion


                                 Job (Work Unit)



             Components of work unit are encapsulated in a mobile agent to
             form a job
             A mobile agent is a software module which moves from node to
             node autonomously and is executed at each node it moves to
             The mobile agent is responsible for the following
                  Migration
                  Communication with Monitoring and Result Server




                                 Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                                14 / 43
Motivation   Focus        Architecture        Implementation          Verification and Validation          Conclusion


                                 Prediction Engine
                                                     Group 2




                Group 1




                                                                                 Group 3




                     Figure: Dedicated Vs Non-dedicated Desktops

                                  Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                                 15 / 43
Motivation      Focus     Architecture        Implementation          Verification and Validation          Conclusion


                                 Prediction Engine




             The peer-to-peer system is used to classify the network into groups
             (g1 , g2 ... gn ).
             Every machine in the group returns the following parameters, in
             response to the resource request query.
                  Resource Availability Factor (raf )
                  Group Availability Factor (ga)




                                  Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                                 16 / 43
Motivation   Focus   Architecture        Implementation          Verification and Validation          Conclusion


               Resource Availability Factor (raf )



     The resource availability factor (raf ) denotes the probability of an
     individual machine’s availability at a given time of the day in a
     week.

     raf
                                               Navailable (t)
                               raf (t) =
                                                Ntotal (t)




                             Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                            17 / 43
Motivation   Focus      Architecture        Implementation          Verification and Validation          Conclusion


                     Group Availability Factor (gaf )



     The group availability quantifies the total group availability at a
     given time of the day in a week (t). This is represented by the
     following:
     gaf

                           ga(t) = currentga (t) − avgb (t)




                                Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                               18 / 43
Motivation   Focus   Architecture         Implementation         Verification and Validation          Conclusion


             Probability of Resource Availability



     For non-dedicated desktops (university lab),

                     Presource      availability   = (raf (t) ∗ ga(t))

     For dedicated desktops (individually owned machines),

                           Presource      availability     = raf (t)




                             Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                            19 / 43
Motivation       Focus    Architecture        Implementation          Verification and Validation          Conclusion




      1      Motivation

      2      Focus

      3      Architecture

      4      Implementation

      5      Verification and Validation

      6      Conclusion



                                  Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                                 20 / 43
Motivation      Focus   Architecture        Implementation          Verification and Validation          Conclusion


                                  Softwares Used


             JDK 1.5
             Java Axis2 webservices
             JXTA P2P Framework
             Aglets agents framework
             C++
             MySQL
             SqlLite
             GNU/Linux and MS Windows XP




                                Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                               21 / 43
Motivation      Focus    Architecture        Implementation          Verification and Validation          Conclusion


                                            Servers



             All the servers host web services using Java Axis2 webservice
             framework
             These webservices enable the following:
                  Registration of job servers
                  Creation of jobs
                  Scheduling Jobs
                  Updating and Retrieving the job status
                  Updating and Retrieving the job result




                                 Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                                22 / 43
Motivation   Focus   Architecture         Implementation           Verification and Validation        Conclusion


                 VCC Client Software - Modules


                         Initialization                    Monitoring Engine




                                                              Mobile Agent
                     Resource Discovery
                                                               Subsystem




                      Broadcast Message                    Broadcast Message
                           Listener                             Receiver




                                     JXTA Peer to Peer Framework




                             Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                            23 / 43
Motivation   Focus         Architecture        Implementation          Verification and Validation          Conclusion


                                          Initialization



                           Bootstrap                                  Join the Peergroup




                     Broadcast the peer                              If the peer is non-
                 information to other peers                       dedicated, join subgroup




                                                                     Initialize the other
                     Start Monitoring Engine
                                                                    modules of the system




                                   Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                                  24 / 43
Motivation   Focus      Architecture           Implementation          Verification and Validation       Conclusion


                              Monitoring Engine




                               C++
             Computer         Module               JNI          JAVA                SqlLite
                          GetSysParameters()                                       Database




                                               <monitors frequency="20000">
                                                        <!--in milliseconds-->
                                                        <monitor name="cpu"/>
                                                        <monitor name="memory"/>
                                               </monitors>




                                Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                               25 / 43
Motivation    Focus          Architecture           Implementation         Verification and Validation         Conclusion


                                    Resource Discovery


                                            Sends availability query
             Queries for resource                                               Peers reply with peer
                                               to each matching
               advertisement                                                   availability factor, if idle
                                                   resource




                              In decreasing of
                           availability, resource                Peers reply if the
                           reservation request is              resource is available
                                    sent




                                            The resource is reserved
                                             and job is dispatched




                                      Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                                     26 / 43
Motivation   Focus        Architecture                Implementation                 Verification and Validation   Conclusion


                         Mobile Agent Subsystem

                                                          Get
                     Aglet Arrived                Server address and
                                                      private key



                                                       Authenticate
                                     Failed




                                                      Start the Job




                                              Poll for User Activity till Job
                                                                                           Dispatch Aglet
                                                       Completion               On User
                                                                                Activity



                                                      Send Results




                                     Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                                        27 / 43
Motivation      Focus     Architecture        Implementation          Verification and Validation          Conclusion


      Broadcaster and Broadcast Message Receiver


             These modules are activated only for non-dedicated peers to share
             the group availability data within their sub group
             Once in a predefined interval (broadcast interval minutes) the
             following data is sent from each client to all the other clients within
             the subgroup:
             <date time>, <peer name>, <peer availability>
             The data broadcasted from the broadcaster module is aggregated in
             each peer and then it is stored in the local grouptaskavg table.
             The grouptaskavg table is used for calculating the avgb (t) value.




                                  Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                                 28 / 43
Motivation   Focus           Architecture         Implementation              Verification and Validation      Conclusion


                                         Job Life Cycle

                       Job is created and        InsertJob webservice to
                                                                                       ScheduleJob
                     stored in a webserver            create the job




                                 Resource selected for               Job pushed to the new
                                    job execution                           resource



                                                            Job
                         Is                              Authentic
                      Machine                                                Failed
                                                           ation
                       Busy?


                                                 Job Starts to execute in
                                                      host machine


                                                  Send Result to result
                                                        Server




                                      Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                                     29 / 43
Motivation       Focus    Architecture        Implementation          Verification and Validation          Conclusion




      1      Motivation

      2      Focus

      3      Architecture

      4      Implementation

      5      Verification and Validation

      6      Conclusion



                                  Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                                 30 / 43
Motivation      Focus    Architecture        Implementation          Verification and Validation          Conclusion


                        Verification and Validation




             Scalability - Real time experiments
             Efficiency of Prediction Algorithm - Simulation




                                 Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                                31 / 43
Motivation   Focus        Architecture          Implementation         Verification and Validation         Conclusion


                                Scalability Testing

             Management           Job Server           Monitoring        Result
               Server                 2                 Server 2        Server 2



             Job Server
                 1                 Resource 1             Resource 4          Resource n-2

                                   Resource 2             Resource 5          Resource n-1
             Monitoring
              Server 1
                                   Resource 3             Resource 6           Resource n

               Result
              Server 1


                                         HFT Cluster Machines
                                                                             n >= 20
                                         Desktop Machine (Windows)
                                                   +
                                         HFT Cluster Workstations




                                  Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                                 32 / 43
Motivation      Focus    Architecture        Implementation          Verification and Validation          Conclusion


                                Test Environment




             18 peers are chosen from computer science Lab 230
             One instance of management server and two instances of job server,
             monitoring server and result server
             The servers and GNU\Linux are chosen from the HFT cluster
                                 Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                                33 / 43
Motivation      Focus    Architecture        Implementation          Verification and Validation          Conclusion


                                          Scenarios

             Generic Scalability Test
                  A combination of 40 jobs (20 from job server 1 and 20 from job
                  server 2) are executed in system and the monitoring server is
                  checked for the status of execution
             Generic Scalability Test - Test Constraints
                  A total of 20 jobs, 15 jobs containing constraints as OS Windows
                  and 5 as GNU\Linux is scheduled in the system
                  The system is tested on whether it is obeying the constraints
             Testing Migration
                  A job is run from the job server into a resource
                  The resource is disturbed by mouse move while execution of the job
                  The system is tested to see whether the job gets migrated to a new
                  machine.


                                 Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                                34 / 43
Motivation      Focus    Architecture        Implementation          Verification and Validation          Conclusion


                 Efficiency of Prediction Algorithms




             Data Collection
             Data Collection Results
             Development of Simulator
             Execution of Scenarios




                                 Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                                35 / 43
Motivation      Focus    Architecture        Implementation          Verification and Validation          Conclusion


                                          Simulator


     Simulation of Prediction models
             Random Scheduling - Simulating the first come first serve model
             Predictive Scheduling - Simulating the predictive algorithm based on
             the group algorithm

     The simulation scripts takes the following main parameters as
     input:
         1   num of jobs - the total number of jobs that has to be scheduled
         2   duration - the time taken for completion of each job
         3   start time - the starting time for the job



                                 Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                                36 / 43
Motivation       Focus          Architecture           Implementation          Verification and Validation          Conclusion


                                                      Results

             Input                                 Sched.            Migrations     Elapsedtime
             (num of jobs, duration, time)         type
                                                   R                 4              35 minutes
             10, 30, 2008-01-12 10:00:00
                                                   P                 0              30 minutes
                                                   R                 10             1:50hrs
             10, 100, 2008-01-12 10:00:00
                                                   P                 2              1:45 hours
                                                   R                 16             3:30hrs
             10, 200, 2008-12-11 10:00:00
                                                   P                 4              3:25hrs
                                                   R                 19             5:10hrs
             10, 300, 2008-12-11 10:00:00
                                                   P                 10             5:10hrs
                                                   R                 55             2:00hrs
             100, 30, 2008-12-11 10:00:00
                                                   P                 48             1:50hrs
                                                   R                 427            8:25hrs
             100, 100, 2008-12-01 12:00:00
                                                   P                 304            7:10hrs
                                                   R                 2              3:25hrs
             100, 100, 2008-12-01 12:00:00
                                                   P                 3              3:30hrs
                                                   R                 42             1 day 3:55hrs
             1000, 100, 2008-12-11 10:00:00
                                                   P                 33             1 day 3:55hrs
                                                   R                 14             13:15hrs
             1000, 45, 2008-12-11 10:00:00
                                                   P                 14             13:15hrs


                    Table: Simulation Results (R-Random, P-Predictive)



                                           Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                                          37 / 43
Motivation      Focus    Architecture        Implementation          Verification and Validation          Conclusion


                                            Results

         1   The predictive scheduling decreases the amount of migrations
             significantly if the num of jobs is lesser than the total number of
             machines in the system
         2   During peak hours, predictive scheduling is useful in selecting
             resources which are less prone to be interrupted by user activity and
             hence reduce the number of migrations
         3   Predictive scheduling also makes a difference with long-duration
             jobs (greater than 45 minutes)
         4   During off-peak hours and when num of jobs is higher than the
             number of machines, the predictive scheduling does not decrease
             the migration by huge factor



                                 Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                                38 / 43
Motivation       Focus    Architecture        Implementation          Verification and Validation          Conclusion




      1      Motivation

      2      Focus

      3      Architecture

      4      Implementation

      5      Verification and Validation

      6      Conclusion



                                  Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                                 39 / 43
Motivation      Focus    Architecture        Implementation          Verification and Validation          Conclusion


                                        Contributions




             Development of a robust desktop grid which takes into account
             multiple job servers, mobility of the job and desktop volatility
             An approach to decentralized resource availability prediction of both
             dedicated and non-dedicated desktops




                                 Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                                40 / 43
Motivation      Focus    Architecture        Implementation          Verification and Validation          Conclusion


                                        Future Work

             Resource Discovery
                  Improvements to JXTA’s basic filtering of resources
                  Studying the system’s performance with more than 20 clients
             Resource Availability Prediction
                  Comparitive study of our model against the model which considers
                  each machine as a dedicated resource
                  Study of efficiency of models in dedicated resources
                  Implementation of advanced mathematical models and probabilistic
                  models in the system.
             Job Programming Model
                  Aglet programming model limits the programmers to develop with
                  global variables to maintain the state.
                  Future work should try to implement programming models like
                  MapReduce over the system and study its efficiency


                                 Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                                41 / 43
Motivation      Focus    Architecture        Implementation          Verification and Validation          Conclusion


                                        Future Work



             Trust
             Fault Tolerance
             Socio-economic Models
             Resource Management
             Leveraging Free Disk Space




                                 Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                                42 / 43
Motivation   Focus   Architecture        Implementation          Verification and Validation          Conclusion




                                       Thank you!




                             Decentralized Resource Availability Prediction in Peer-to-Peer Desktop Grids
                                                                                                            43 / 43

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:8
posted:8/9/2011
language:English
pages:43