Architecture and Design of Distributed Database Systems

Document Sample
Architecture and Design of Distributed Database Systems Powered By Docstoc
					            Architecture and Design of
           Distributed Database Systems
              WAMDM Cloud Computing Group
                             Haiping Wang
                               2010-05-08


2012/2/9              WAMDM Cloud Group     1
                         Outline
• Architecture
      – Client/Server, P2P
      – DDBMS,MDBS
• Design
      – Strategies
      – Issues
      – Fragmentation
      – Allocation
• Conclusion
2012/2/9                     WAMDM Cloud Group   2
   Client/Server Reference Architecture
                 • Begin at 1990’s
                 • Communication at the level of SQL
                   statement
                 • Client
                      – Application , user interface, DBMS Client
                        responsible for cache management( cached
                        data and cached transaction locks)
                      – Possible for consistency checking of user
                        queries(not common)
                 • Server
                      – Query processing, optimization,
                        transaction and storage management
                      – One server
                            • Comparison with centralized DB
                      – Multiple Server
                            • client manages its own connection(heavy client)
                            • Client only know it “home server”(light client)




2012/2/9         WAMDM Cloud Group                                              3
            Physical Data Organization in P2P
                  Distributed Systems
• Data usually fragmented and
  replicated
• Three layer organization
   – LIS (local internal schema)
   – LCS(local conceptual schema)
   – GCS ( global conceptual
     schema)
   – ESs (external schema), for user
     applications and access


 2012/2/9                   WAMDM Cloud Group   4
 Functional Schematic of an Integrated
                DDBMS


                                For data independence
                                Extended the ANSI/SPARC
                                GD/D for global mapping
                                LD/D for local mapping




2012/2/9        WAMDM Cloud Group                     5
               Components of a DDBMS
• User processor
      – User interface handler
      – Semantic data controller
      – Global query optimizer and
        decomposer
      – Distributed execution monitor
• Data processor
      – Local query optimizer
      – Local recovery manager
      – Run-time support processor
•

    2012/2/9                    WAMDM Cloud Group   6
                  MDBS Architecture
With a GCS                             Without GCS




GCS,GES may use different data model
and language                                    Local system layer
Homogeneous:
     Unilingual(example:MULITIBASE)
                                                Multidatabase layer
      Multilingual(permit each user to
     access the global data by means of
     external schema)
heterogeneous
  2012/2/9                            WAMDM Cloud Group                7
Difference between MDBS and DDBMS
Definition difference      DDBMS                         MDBs
Global conceptual schema   The conceptual view of the     The collection of some of
(fundamental difference)   entire database, mapping      the local DB that wants to
                           the global schema to local    share, mapping the local
                           conceptual schema             conceptual schema to a
                                                         global schema

Global database            Equal to the union of local   A subset of the union of
                           databases                     the local databases
Design strategy            Usually designed by top-      Usually designed by
                           down process                  bottom-up process




2012/2/9                         WAMDM Cloud Group                                    8
                         Outline
• Architecture
      – Client/Server, P2P
      – DDBMS,MDBS
• Design
      – Strategies
      – Issues
      – Fragmentation
      – Allocation
• Conclusion
2012/2/9                     WAMDM Cloud Group   9
           Framework of Distribution
                              • Level of sharing
                                    – Share nothing
                                    – Share data
                                    – Share data+ program
                              • Access pattern
                                    – Static
                                    – Dynamic
                              • Level of knowledge on
                                access pattern
                                    – Know nothing
                                    – The designer know partial
                                      information
                                    – The designer know complete
                                      information

2012/2/9            WAMDM Cloud Group                          10
           Top-down design process
• An iteration of five steps:
      – Requirement analysis
      – Conceptual design and
        view integration
         E-R representation and
        translation to RDB schema
      – Distribution design
          Data fragmentation and
        allocation
      – Physical design
      – Tuning

2012/2/9                     WAMDM Cloud Group   11
           Bottom-up design process
• Primarily for multidatabase applications
• Terms related
      – Data integration/fusion
      – P2P
• Not covered by this course




2012/2/9                 WAMDM Cloud Group   12
           Distribution Design Issues
•   Why to fragment
•   How to fragment
•   Correctness of fragmentation
•   How to allocate
•   The information for fragmentation




2012/2/9             WAMDM Cloud Group   13
           Reasons for fragmentation

   To enhance intra-query concurrency
   To increase the throughput

     But extra cost for queries involving
     more than one segment residing at
     different sites


2012/2/9             WAMDM Cloud Group      14
Correctness for rules of fragmentation




2012/2/9       WAMDM Cloud Group     15
   Horizontal Partition relation PROJ




2012/2/9        WAMDM Cloud Group       16
           Horizontal fragmentation
• Primary horizontal fragmentation – to partition a
  relation by using predicates on that relation

• Derived horizontal fragmentation – to partition a
  relation by using predicates on another relation

• Two aspects affecting fragmentation
      – data information, and
      – application information

2012/2/9                  WAMDM Cloud Group           17
           Data information




2012/2/9        WAMDM Cloud Group   18
            Application information
• Both qualitative and quantitative are needed
• Definition
      – simple predicate
           example: LOC= “New York”
      – Minterm predicate




2012/2/9                    WAMDM Cloud Group    19
           Application information

 Minterm selectivity – sel(mi)
 The number of tuples of a relation returned by a
 query specified by the minterm mi


  Access frequency – acc(mi)
  The frequency with which user applications access
  data using a query specified by the minterm mi



2012/2/9               WAMDM Cloud Group              20
   Primary horizontal fragmentation




2012/2/9        WAMDM Cloud Group     21
     The first step for primary horizontal
                 fragmentation
• To find a set of simple predicates with the
  properties of Completeness and Minimal
• Completeness
       A set of predicate Pr is complete iff there is an
    equal probability of access by any application to any
    two tuples belonging to any minterm fragment that
    is defined according to Pr.



2012/2/9                 WAMDM Cloud Group                  22
           Minimal




2012/2/9   WAMDM Cloud Group   23
           Steps for primary horizontal
                  fragmentation
• Use COM_MIN algorithm to generate a
  complete and minimal set of predicates Pr’
  given a set of simple predicates
• Derive the set of minterm predicates
• Elimination of some of the meaningless
  minterm fragments



2012/2/9             WAMDM Cloud Group         24
                      COM_MIN
Rule1 fundamental rule of completeness and minimality :
a relation or fragment is partitioned “ into at least two
parts which are accessed differently by at least one
application”




Input: R: relation; Pr: set of simple predicates
Output: Pr’: set of simple predicates
Function: generate a complete and minimal set of predicates Pr’
given a set of simple predicates
2012/2/9                 WAMDM Cloud Group                        25
           COM_MIN




2012/2/9    WAMDM Cloud Group   26
           PHORIZONTAL




2012/2/9      WAMDM Cloud Group   27
           Example for primary horizontal
                  fragmentation




2012/2/9              WAMDM Cloud Group     28
           Example for primary horizontal
                  fragmentation




2012/2/9              WAMDM Cloud Group     29
           Example for primary horizontal
                  fragmentation

• Use COM_MIN algorithm to get a complete
  and minimal simple predicates
              Pr'   p1 , p5 , p2 , p3 , p4 




2012/2/9                     WAMDM Cloud Group   30
           Example for primary horizontal
                  fragmentation




2012/2/9              WAMDM Cloud Group     31
           Example for primary horizontal
                  fragmentation




2012/2/9              WAMDM Cloud Group     32
           Example for primary horizontal
                  fragmentation




2012/2/9              WAMDM Cloud Group     33
           Example for primary horizontal
                  fragmentation




2012/2/9              WAMDM Cloud Group     34
   Derived horizontal fragmentation




2012/2/9        WAMDM Cloud Group     35
           Vertical fragmentation
                                 SELECT BUDGET
                                 FROM PROJ
                                 WHERE PNO=Value
                                 SELECT PNAME
                                 FROM PROH
                                 WHERE LOC=Value
                                 SELECT PNAME,BUDGET
                                 FROM PROJ
                                 SELECT SUM(BUDGET)
                                 FROM PROJ
                                 WHERE LOC=Value
2012/2/9           WAMDM Cloud Group                   36
           Factors to be considered
• Attributes usage(AUM)
• Attributes affinity(AAM)
      – AAM :affinity between two attributes
      – AM: used for grouping the attributes of a relation
        based on AAM(suggested in 1975,1984 )
      – bond
• How to fragment by attributes


2012/2/9                  WAMDM Cloud Group                  37
                               AU Matrix
                  1 if     attribute AJ             is referenced by query qi
  use(qi , Aj )  
                  0                                 otherwise
                                 A1      A2     A3        A4
                          q1      1      0 1 0
                          q2      0      1 1 0
                                                 
                          q3      0      1 0 1
                                                 
                          q4      0      0 1 1
                                          ( AUM )

aff ( Ai , Aj )                                               ref (q )acc (q )
                    k |use ( qk , Ai ) 1use ( qk , A j ) 1 PAYl
                                                                      l   k   l        k


 2012/2/9                             WAMDM Cloud Group                           38
                                  AA Matrix
            acc1 (q1 )  15 acc2 ( q1 )  20 acc3 ( q1 )  10
            acc1 (q2 )  5 acc2 (q2 )  0 acc3 (q2 )  0
           acc1 (q3 )  25 acc2 (q3 )  25 acc3 ( q3 )  25
             acc1 (q4 )  3 acc2 (q4 )  0 acc3 (q4 )  0
                                                           A1   A2   A3   A4
     A1     A2   A3   A4
                                               A1      45 0 45 0 
q1     1    0 1 0        sites  3                   0 80 5 75
       0    1 1 0                            A2
q2                       ref l (qk )  1                        
q3     0    1 0 1                            A3      45 5 53 3 
                                                                
q4     0    0 1 1
                                               A4      0 75 3 78
             ( AUM )
                                                           ( AAM )

2012/2/9                               WAMDM Cloud Group                       39
                  AM(global affinity measure)
             n    n
AM   aff ( Ai , Aj )[aff ( Ai , Aj 1 )  aff ( Ai , A j 1 )  aff ( Ai 1 , A j )  aff ( Ai 1 , A j )]
         i 1 j 1

where
aff ( A0 , Aj )  aff ( Ai , A0 )  aff ( An 1 , Aj )  aff ( Ai , An 1 )  0

       A1        A2   A3   A4                                              AA is symmetric
  A1    45 0 45 0 
 A2     0 80 5 75
                   
 A3     45 5 53 3                           n    n
 A4
       
        0 75 3 78
                                   AM   aff ( Ai , Aj )[aff ( Ai , Aj 1 )  aff ( Ai , Aj 1 )]
            ( AAM )                          i 1 j 1

                                   aff ( A0 , Aj )  aff ( Ai , An 1 )  0
  2012/2/9                                   WAMDM Cloud Group                                        40
            AM(global affinity measure)
                       n     n
             AM   aff ( Ai , A j )[aff ( Ai , A j 1 )  aff ( Ai , A j 1 )]
                      i 1 j 1




            n
             n                                   n
                                                                                   
 AM    aff ( Ai , Aj )aff ( Ai , Aj 1 )   aff ( Ai , Aj )aff ( Ai , Aj 1 ) 
      j 1  i 1                              i 1                                

                                   n
           bond ( Ax , Ay )   aff ( Az , Ax )aff ( Az , Ay )
                                  z 1



                            n
                 AM   [bond ( Aj , Aj 1 )  bond ( Aj , Aj 1 )]
                           i 1

2012/2/9                                 WAMDM Cloud Group                             41
                   Bond energy algorithm
      input: AA: attributes affinity matrix
      output: CA: clustered affinity matrix
      Function: determine groups of similar items
      Steps:
          1.Initialization,place and fix two of the columns of AA into CA
          2. Iteration. Pick each of the remaining n-i columns and find the proper
      palace to insert the column to CA, By compute the cont
          3. Row ordering. Make CA to be symmetrically


           Note:
              The final group are insensitive to the order
              The computation time is O(n*n)



2012/2/9                             WAMDM Cloud Group                          42
                                            Cont
                            A1 A2 ... Ai 1 Ai Aj A j 1... An
                                   AM '                        AM ''


  AMold  AM '  AM ''  bond ( Ai1, Ai )  bond ( Ai , Aj )  bond ( Aj , Ai )  bond ( Aj , Aj 1)

AMnew  AM  AM  bond ( Ai1, Ai )  bond ( Aj , Aj1)  2bond ( Ai , Ak )  2bond ( Ak , Aj )
               '       '



   cont ( Ai , Ak , Aj )  AM new  AM old
    2bond ( Ai , Ak )  2bond ( Ak , Aj )  2bond ( Ai , Aj )
    2012/2/9                              WAMDM Cloud Group                                   43
             Calculation of CA Matrix
             A1   A2                                  A1   A3   A2
       A1    45 0                          A1       45 45 0        
       A2    0 80                          A2       0 5 80         
                                                                   
       A3    45 5                          A3       45 53 5        
                                                                   
       A4    0 75                          A4       0 3 75         
                 (a)                                      (b)
             A1   A3   A2   A4                       A1    A3   A2   A4
       A1    45 45 0 0                    A1        45 45 0 0 
       A2    0 5 80 75                    A3        45 53 5 3 
                                                              
       A3    45 53 5 3                    A2        0 5 80 75
                                                              
       A4    0 3 75 78                    A4        0 3 75 78
                 (c )                                     (d )
2012/2/9                         WAMDM Cloud Group                        44
                 Definition for Partition
    AQ(qi )  { Aj | use(qi , Aj )  1}            CQ          ref
                                                              qi Q S j
                                                                                j   (qi )acc j (qi )

 TQ  {qi | AQ(qi )  TA}                          CTQ            ref             j       (qi )acc j (qi )
    BQ  {qi | AQ(qi )  BA}                                    qi TQ S j


 OQ  Q  {TQ             BQ}                      CBQ            ref
                                                                qi BQ S j
                                                                                         j   (qi )acc j (qi )

                                                   COQ             ref                j   (qi )acc j (qi )
    z  CTQ * CBQ  COQ 2                                        qi OQ S j


                                                                  A1       A3       A2         A4
• Two complications                                      A1      45 45 0 0 
       – Splitting (2-way, m-way, 2m )                           45 53 5 3 
       – The location of the block of attributes         A3                
         that should form one fragment                   A2      0 5 80 75
•     Shift                                                                
                                                         A4      0 3 75 78

2012/2/9                             WAMDM Cloud Group                                                          45
                             Caculate z
   A1 A3   A2 A4              acc1 (q1 )  15 acc2 ( q1 )  20 acc3 ( q1 )  10
                               acc1 (q2 )  5 acc2 (q2 )  0 acc3 (q2 )  0
   TA  { A1 , A3 }
                              acc1 (q3 )  25 acc2 (q3 )  25 acc3 ( q3 )  25
   BA  { A2 , A4 }
                               acc1 (q4 )  3 acc2 (q4 )  0 acc3 (q4 )  0
  Q  {q1 , q2 , q3 , q4 }
   AQ(q1 )  { A1 , A3 }        sites  3
   AQ(q2 )  { A2 , A3 }        ref l (qk )  1
   AQ(q3 )  { A2 , A4 }
                               CQ  15  20  10  5  25  25  25  3  128
   AQ(q4 )  { A3 , A4 }
                                            CTQ  15  20  10  45
      TQ  {q1}
                                            CBQ  25  25  25  75
   BQ  {q3 }                                       COQ  5  3  7
  OQ  {q2 , q4 }                             z  45*75  7 2  3326
2012/2/9                        WAMDM Cloud Group                                 46
               Partition algorithm
 Input: CA: clustered affinity matrix; R: relation;
        ref: attribute usage matrix; acc: access frequency matrix
 Output: F: set of fragments
 Function: part the relation R using Vertical fragmentation
 Steps:
     1. determine the z value for the first column
     2. determine the best partition
     3. do the partition operation




2012/2/9                   WAMDM Cloud Group                    47
           Hybrid fragmentation
              Tree Structured partition

                                                  bigtable

                                              H         H         H



                               tablet-1           tablet-2            tablet-3
                                 V   V              V       V           V   V

                              CF1       CF2       CF1       CF2       CF1   CF2




2012/2/9            WAMDM Cloud Group                                           48
                       Allocation

            fragments : F  F1 , F2 ,..., Fn 

              sites:S  S1 , S 2 ,..., S m 

              queries:Q  q1 , q2 ,..., qq 


      Finding the “optimal” distribution of F to S ?


2012/2/9                   WAMDM Cloud Group           49
                      Optimally
• Minimal cost
      – Storage cost
      – Query cost
      – Update cost
      – Data communication
• Performance
      – Minimize the response time
      – Maximize the system throughput at each site

2012/2/9                 WAMDM Cloud Group            50
           Assumptions and Definitions
• A single fragment Fk
• Query read  only:T  t1 , t2 ,..., tm 

               update:U  u1 , u2 ,..., um 
• Communication cost
       C T   {c12 , c13 ,..., c1m ,..., cm 1,m }
             C (U )  {c , c ,...,c ,...,c
               '         '
                         12
                               '
                               13
                                          '
                                          1m
                                                  '
                                                  m1,m   }
• Storage cost D  {d1 , d 2 ,..., d m }
2012/2/9                      WAMDM Cloud Group               51
           Assumptions and Definitions
                                   I S

         1 if   the   faragment    Fk    is assigned to site S j
    xj  
         0                           otherwise


           m                                                     
       min  (  x j u j cij  t j min cij ) 
                           '
                                                        Ix j d j 
                                    j | S j I
            i 1 j|S j I
                                                      j|S j     
                                                                  

2012/2/9                    WAMDM Cloud Group                         52
           Not suitable for DDB design
• NP-complete
• Reasons
      – Cannot treat fragments as individual
      – No consideration of integrity enforcement
      – No consideration of concurrency control
        mechanisms




2012/2/9                 WAMDM Cloud Group          53
                Information required
• Database info.
     – Number of tuples: card ( F j )
     – Fragment size: size ( F j )  card ( F j ) * length( F j )
• Application info. URij & RRij
• Site info: USCk & LPCk
• Network info: cost per frame g ij



2012/2/9                       WAMDM Cloud Group                    54
                   Allocation model
     1 if   the    faragment       Fk     is assigned to site S j
xj  
     0                                  otherwise



           TOC       QPC   
                    qi Q
                                     i
                                           Sk S F j F
                                                            STC jk



              STC jk  USCk * size( F j ) * x jk

2012/2/9                     WAMDM Cloud Group                       55
                             Allocation model
                                 QPCi  PCi  TCi
                                PCi  ACi  IEi  CCi
           ACi            (u
                       Sk S F j F
                                            ij   *URij  rij * RRij )* x jk * LPCk

                                    TCi  TCU i  TCRi

    TCUi             u
                 Sk F Fj F
                                     ij   * xij * go (i ),k      u
                                                                Sk S Sk F
                                                                                ij   *x jk * g k ,o (i )

                                                                 sel ( Fj )* length( Fj )
TCRi       
           F j F
                     min(rij * x jk * go (i ),k  rij * x jk *
                     Sk S                                                fsize
                                                                                               * g k ,o ( i ) )
2012/2/9                                    WAMDM Cloud Group                                            56
               Constrains
• Execution time of qi  maximum response
  time of q i
• Storage constrains
• Processing constrains




2012/2/9          WAMDM Cloud Group         57
                  Solution
• assume all candidate partitions known, select
  the “best ” partitioning
• ignore replication at first
• sliding window on fragments




2012/2/9           WAMDM Cloud Group              58
               Conclusion
• Three architecture of DDBMS
• Two fragment methods
• The cost of allocation




2012/2/9          WAMDM Cloud Group   59
           Thank you!!!

2012/2/9       WAMDM Cloud Group   60

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:40
posted:2/10/2012
language:Latin
pages:60