A Universal Calculus for Stream Processing Languages

Document Sample
A Universal Calculus for Stream Processing Languages Powered By Docstoc
					                                                                 1




      A Universal Calculus for
      Stream Processing Languages
       Robert Soulé, Martin Hirzel, Robert Grimm, Buğra Gedik,
       Henrique Andrade, Vibhore Kumar, and Kun-Lung Wu

       New York University and IBM Research



Thursday, March 18, 2010
                           2


      Stream Processing
      Is Everywhere




Thursday, March 18, 2010
                           2


      Stream Processing
      Is Everywhere




Thursday, March 18, 2010
                           2


      Stream Processing
      Is Everywhere




Thursday, March 18, 2010
                           2


      Stream Processing
      Is Everywhere




Thursday, March 18, 2010
                                                    2


      Stream Processing
      Is Everywhere



                  Move large amounts of data
              through several computational steps




Thursday, March 18, 2010
                           3


      Stream Processing
      Has Many Flavors




Thursday, March 18, 2010
                           3


      Stream Processing
      Has Many Flavors

             StreamIt:
           synchronous
            processing




Thursday, March 18, 2010
                                           3


      Stream Processing
      Has Many Flavors

             StreamIt:         CQL:
           synchronous     SQL + streams
            processing




Thursday, March 18, 2010
                                                     3


      Stream Processing
      Has Many Flavors

             StreamIt:                   CQL:
           synchronous               SQL + streams
            processing
                               Sawzall:
                           functional data
                             processing
                             in the large
Thursday, March 18, 2010
                                                     4


      Stream Processing
      Has Many Implementations

             StreamIt:                   CQL:
           synchronous               SQL + streams
            processing
                               Sawzall:
                           functional data
                             processing
                             in the large
Thursday, March 18, 2010
                                                                                       5




      Variety Breeds Confusion
            We want to understand and compare streaming languages

                  What is their expressiveness?

                  How to optimize the data processing steps?

                  How to scale the overall applications? Especially across clusters?

            Enter our universal calculus: Brooklet

                  Formal foundation for answering the above questions

                  Provably correct optimizations and translations



Thursday, March 18, 2010
                                                                             6




      Outline of This Talk

            Motivation

            Requirements

            The Brooklet Core Calculus

            Generality: Translating StreamIt, CQL, and Sawzall to Brooklet

            Utility: Optimizing Brooklet to Brooklet

            Outlook and Conclusions




Thursday, March 18, 2010
                                              7




      Elements of a Streaming App

                             State




                  Queue               Queue
                           Operator

Thursday, March 18, 2010
                                    8




      Elements of a Streaming App




Thursday, March 18, 2010
                                               9




      Elements of a Streaming App




                           Not all operators
                              have state




Thursday, March 18, 2010
                                           10




      Elements of a Streaming App




                           Operators may
                            share state




Thursday, March 18, 2010
                                  11




      Requirements for Calculus




Thursday, March 18, 2010
                                             11




      Requirements for Calculus
                                   Make
                                  explicit




Thursday, March 18, 2010
                                             11




      Requirements for Calculus
                                   Make
                                  explicit




                             Make
                            explicit
                           and 1-to-1




Thursday, March 18, 2010
                                                             11




      Requirements for Calculus
                               Make
                            Make non-              Make
                           trigger non-
                           deterministic          explicit
                           deterministic




                                             Make
                                            explicit
                                           and 1-to-1




Thursday, March 18, 2010
                                                             11




      Requirements for Calculus
                               Make
                            Make non-              Make
                           trigger non-
                           deterministic          explicit
                           deterministic




           Treat                             Make
        functions as                        explicit
          opaque                           and 1-to-1




Thursday, March 18, 2010
                                                                    12




      Brooklet Syntax

                                           $total




                             trades                    volume

                                          Sum


                           (volume, $total) ← Sum(trades, $total)


Thursday, March 18, 2010
                                                             13




      Function Environment

                                       $total




                           trades                  volume

                                      Sum
                                      Sum


                           F: The function implementations


Thursday, March 18, 2010
                                                              14




      Queue Store

                                        $total




                           trades                   volume

                           trades      Sum           volume




                             Q: The contents of the queues


Thursday, March 18, 2010
                                                               15




      Variable Store

                                         $total
                                         $total




                           trades                    volume

                                        Sum


                            V: The contents of the variables


Thursday, March 18, 2010
                                                                16


      Brooklet
      Operational Semantics

                                            $total




                           trades                      volume

                                           Sum


                                    F   <Q, V> → <Q’, V’>


Thursday, March 18, 2010
                                                                                                       17




   Complete Calculus
  4                  e
          Robert Soul´ et al.

  Brooklet syntax:                                  Brooklet semantics: Fb ￿ ￿V, Q￿ −→ ￿V ￿ , Q￿ ￿
  Pb ::= out in op           Brooklet program                 d, b = Q(qi )
  out ::= output q ;      Output declaration           op = (_, _) ← f (q, v);
  in ::= input q ;          Input declaration          ￿   ￿
                                                     (b , d ) = Fb (f )(d, i, V (v))
  op ::= ( q, v ) ← f ( q, v );      Operator       V ￿ = updateV (op, V, d )
                                                                               ￿
  q   ::= id                 Queue identifier                                       ￿
                                                    Q￿ = updateQ(op, Q, qi , b )
  v   ::= $ id             Variable identifier                                        (E-FireQueue)
  f   ::= id              Function identifier          Fb ￿ ￿V, Q￿ −→ ￿V ￿ , Q￿ ￿
  Brooklet example: IBM market maker.                     op = (_, v) ← f (_, _);
  output result;                                                                              (E-UpdateV)
                                                    updateV (op, V, d) = [v ￿→ d]V
  input bids, asks;
  (ibmBids) ←    SelectIBM(bids);                          op = (q, _) ← f (_, _);
  (ibmAsks) ←    SelectIBM(asks);                              df , bf = Q(qf )
  ($lastAsk)←    Window(ibmAsks);                             Q￿ = [qf ￿→ bf ]Q
  (ibmSales)←    SaleJoin(ibmBids,$lastAsk);        Q￿￿   = [∀qi ∈ q : qi ￿→ Q(qi ), bi ]Q￿
  (result,$cnt) ←    Count(ibmSales,$cnt);             updateQ(op, Q, qf , b) = Q     ￿￿
                                                                                              (E-UpdateQ)


                               Fig. 1. Brooklet syntax and semantics.

     3.1 Brooklet Program Example: IBM Market Maker
     As an example of a streaming program, we consider a hypothetical application
Thursday, March 18, 2010
                                                                      18


      Example:
      A Fannie Mae Bid/Ask Join

                    as          $lastAsk            $total
                           ks

                                           trades            volume
                                SaleJoin            Sum

                 bi ds

                                $lastBid



Thursday, March 18, 2010
                                                                                 19


      Example:
      A Fannie Mae Bid/Ask Join
                                $lastAsk = <FNM,0,∞>       $total = 0

  <FNM,1,€2>                        $lastAsk                $total
                    as
                           ks

                                                  trades                volume
                                    SaleJoin                Sum

                 bi ds

  <FNM,1,€1>                        $lastBid
  <FNM,1,€2>
                                $lastBid = <FNM,0,0>


Thursday, March 18, 2010
                                                                                 20


      Example:
      A Fannie Mae Bid/Ask Join
                                $lastAsk = <FNM,0,∞>       $total = 0

  <FNM,1,€2>                        $lastAsk                $total
                    as
                           ks

                                                  trades                volume
                                    SaleJoin                Sum

                 bi ds

  <FNM,1,€1>                        $lastBid
  <FNM,1,€2>
                                $lastBid = <FNM,0,0>


Thursday, March 18, 2010
                                                                                 20


      Example:
      A Fannie Mae Bid/Ask Join
                                $lastAsk = <FNM,0,∞>       $total = 0

  <FNM,1,€2>                        $lastAsk                $total
                    as
                           ks

                                                  trades                volume
                                    SaleJoin                Sum
                                    <FNM,1,€1>
                 bi ds

                                    $lastBid
  <FNM,1,€2>
                                $lastBid = <FNM,0,0>


Thursday, March 18, 2010
                                                                                 20


      Example:
      A Fannie Mae Bid/Ask Join
                                $lastAsk = <FNM,0,∞>       $total = 0

  <FNM,1,€2>                        $lastAsk                $total
                    as
                           ks

                                                  trades                volume
                                    SaleJoin                Sum

                 bi ds

                                    $lastBid
  <FNM,1,€2>                               <FNM,1,€1>
                                $lastBid = <FNM,0,0>


Thursday, March 18, 2010
                                                                                21


      Example:
      A Fannie Mae Bid/Ask Join
                                $lastAsk = <FNM,0,∞>      $total = 0

  <FNM,1,€2>                        $lastAsk               $total
                    as
                           ks

                                                 trades                volume
                                    SaleJoin               Sum

                 bi ds

  <FNM,1,€2>                        $lastBid

                            $lastBid = <FNM,1,€1>


Thursday, March 18, 2010
                                                                                22


      Example:
      A Fannie Mae Bid/Ask Join
                                $lastAsk = <FNM,0,∞>      $total = 0

  <FNM,1,€2>                        $lastAsk               $total
                    as
                           ks

                                                 trades                volume
                                    SaleJoin               Sum

                 bi ds

  <FNM,1,€2>                        $lastBid

                            $lastBid = <FNM,1,€1>


Thursday, March 18, 2010
                                                                                23


      Example:
      A Fannie Mae Bid/Ask Join
                                $lastAsk = <FNM,0,∞>      $total = 0

  <FNM,1,€2>                        $lastAsk               $total
                    as
                           ks

                                                 trades                volume
                                    SaleJoin               Sum

                 bi ds

  <FNM,1,€2>                        $lastBid

                            $lastBid = <FNM,1,€1>


Thursday, March 18, 2010
                                                                                23


      Example:
      A Fannie Mae Bid/Ask Join
                                $lastAsk = <FNM,0,∞>      $total = 0

                    as              $lastAsk               $total
                           ks

                                   <FNM,1,€2>    trades                volume
                                    SaleJoin               Sum

                 bi ds

  <FNM,1,€2>                        $lastBid

                            $lastBid = <FNM,1,€1>


Thursday, March 18, 2010
                                                                                 23


      Example:
      A Fannie Mae Bid/Ask Join
                                $lastAsk = <FNM,0,∞>       $total = 0
                                           <FNM,1,€2>

                    as              $lastAsk                $total
                           ks

                                                  trades                volume
                                    SaleJoin                Sum

                 bi ds

  <FNM,1,€2>                        $lastBid

                            $lastBid = <FNM,1,€1>


Thursday, March 18, 2010
                                                                              24


      Example:
      A Fannie Mae Bid/Ask Join
                            $lastAsk = <FNM,1,€2>       $total = 0

                    as           $lastAsk                $total
                           ks

                                               trades                volume
                                SaleJoin                 Sum

                 bi ds

  <FNM,1,€2>                     $lastBid

                            $lastBid = <FNM,1,€1>


Thursday, March 18, 2010
                                                                              25


      Example:
      A Fannie Mae Bid/Ask Join
                            $lastAsk = <FNM,1,€2>       $total = 0

                    as           $lastAsk                $total
                           ks

                                               trades                volume
                                SaleJoin                 Sum

                 bi ds

  <FNM,1,€2>                     $lastBid

                            $lastBid = <FNM,1,€1>


Thursday, March 18, 2010
                                                                              26


      Example:
      A Fannie Mae Bid/Ask Join
                            $lastAsk = <FNM,1,€2>       $total = 0

                    as           $lastAsk                $total
                           ks

                                               trades                volume
                                SaleJoin                 Sum

                 bi ds

  <FNM,1,€2>                     $lastBid

                            $lastBid = <FNM,1,€1>


Thursday, March 18, 2010
                                                                              26


      Example:
      A Fannie Mae Bid/Ask Join
                            $lastAsk = <FNM,1,€2>       $total = 0

                    as           $lastAsk                $total
                           ks

                                               trades                volume
                                SaleJoin                 Sum
                                <FNM,1,€2>
                 bi ds

                                 $lastBid

                            $lastBid = <FNM,1,€1>


Thursday, March 18, 2010
                                                                              27


      Example:
      A Fannie Mae Bid/Ask Join
                            $lastAsk = <FNM,1,€2>       $total = 0

                    as           $lastAsk                $total
                           ks

                                               trades                volume
                                SaleJoin                 Sum
                                <FNM,1,€2>
                 bi ds

                                 $lastBid

                            $lastBid = <FNM,1,€1>


Thursday, March 18, 2010
                                                                              27


      Example:
      A Fannie Mae Bid/Ask Join
                            $lastAsk = <FNM,1,€2>       $total = 0

                    as           $lastAsk                $total
                           ks

                                               trades                volume
                                SaleJoin                 Sum
                                <FNM,1,€2>
                 bi ds

                                 $lastBid
                                       <FNM,1,€2>
                            $lastBid = <FNM,1,€1>


Thursday, March 18, 2010
                                                                              28


      Example:
      A Fannie Mae Bid/Ask Join
                            $lastAsk = <FNM,1,€2>       $total = 0

                    as           $lastAsk                $total
                           ks

                                               trades                volume
                                SaleJoin                 Sum
                                <FNM,1,€2>
                 bi ds

                                 $lastBid

                            $lastBid = <FNM,1,€2>


Thursday, March 18, 2010
                                                                              29


      Example:
      A Fannie Mae Bid/Ask Join
                            $lastAsk = <FNM,1,€2>       $total = 0

                    as           $lastAsk                $total
                           ks

                                               trades                volume
                                SaleJoin                 Sum
                                <FNM,1,€2>
                 bi ds

                                 $lastBid

                            $lastBid = <FNM,1,€2>


Thursday, March 18, 2010
                                                                                29


      Example:
      A Fannie Mae Bid/Ask Join
                            $lastAsk = <FNM,1,€2>         $total = 0

                    as           $lastAsk                  $total
                           ks
                                             <FNM,1,€2>
                                               trades                  volume
                                SaleJoin                   Sum

                 bi ds

                                 $lastBid

                            $lastBid = <FNM,1,€2>


Thursday, March 18, 2010
                                                                               29


      Example:
      A Fannie Mae Bid/Ask Join
                            $lastAsk = <FNM,1,€2>        $total = 0

                    as           $lastAsk                 $total
                           ks

                                               trades   <FNM,1,€2>    volume
                                SaleJoin                  Sum

                 bi ds

                                 $lastBid

                            $lastBid = <FNM,1,€2>


Thursday, March 18, 2010
                                                                              30


      Example:
      A Fannie Mae Bid/Ask Join
                            $lastAsk = <FNM,1,€2>       $total = 0

                    as           $lastAsk                $total
                           ks

                                               trades   <FNM,1>      volume
                                SaleJoin                 Sum

                 bi ds

                                 $lastBid

                            $lastBid = <FNM,1,€2>


Thursday, March 18, 2010
                                                                              30


      Example:
      A Fannie Mae Bid/Ask Join
                                                        <FNM,1>
                            $lastAsk = <FNM,1,€2>       $total = 0

                    as           $lastAsk                $total
                           ks

                                               trades   <FNM,1>      volume
                                SaleJoin                 Sum

                 bi ds

                                 $lastBid

                            $lastBid = <FNM,1,€2>


Thursday, March 18, 2010
                                                                              31


      Example:
      A Fannie Mae Bid/Ask Join
                            $lastAsk = <FNM,1,€2>       $total = 1

                    as           $lastAsk                $total
                           ks

                                               trades   <FNM,1>      volume
                                SaleJoin                 Sum

                 bi ds

                                 $lastBid

                            $lastBid = <FNM,1,€2>


Thursday, March 18, 2010
                                                                              32


      Example:
      A Fannie Mae Bid/Ask Join
                            $lastAsk = <FNM,1,€2>       $total = 1

                    as           $lastAsk                $total
                           ks

                                               trades   <FNM,1>      volume
                                SaleJoin                 Sum

                 bi ds

                                 $lastBid

                            $lastBid = <FNM,1,€2>


Thursday, March 18, 2010
                                                                               32


      Example:
      A Fannie Mae Bid/Ask Join
                            $lastAsk = <FNM,1,€2>       $total = 1

                    as           $lastAsk                $total
                           ks
                                                                     <FNM,1>
                                               trades                volume
                                SaleJoin                 Sum

                 bi ds

                                 $lastBid

                            $lastBid = <FNM,1,€2>


Thursday, March 18, 2010
                                                                 33




      Translations
       Demonstrating Brooklet’s generality
       by translating three rather diverse streaming languages




Thursday, March 18, 2010
                                                                            34


      CQL, StreamIt, Sawzall:
      One Translation Approach
                                       Expose graph topology


                                                        Expose implicit
                                                       and explicit state

               Wrap original operators in
                higher-order functions



                           Functions     <   Queues     ,   Variables   >


Thursday, March 18, 2010
                                                                                34


      CQL, StreamIt, Sawzall:
      One Translation Approach
                                                                      Make
                                                                  queues explicit
                                       Expose graph topology
                                                                    and 1-to-1

                                                        Expose implicit
                                                       and explicit state

               Wrap original operators in
                higher-order functions



                           Functions     <   Queues     ,   Variables   >


Thursday, March 18, 2010
                                                                                 34


      CQL, StreamIt, Sawzall:
      One Translation Approach
                                                                      Make
                                                                  queues explicit
                                       Expose graph topology
                                                                    and 1-to-1

                                                        Expose implicit
                                                       and explicit state

               Wrap original operators in                              Make
                higher-order functions                              state explicit


                           Functions     <   Queues     ,   Variables   >


Thursday, March 18, 2010
                                                                                 34


      CQL, StreamIt, Sawzall:
      One Translation Approach
                                                                      Make
                                                                  queues explicit
                                       Expose graph topology
           Do not                                                   and 1-to-1
         model local
        computations                                    Expose implicit
                                                       and explicit state

               Wrap original operators in                              Make
                higher-order functions                              state explicit


                           Functions     <   Queues     ,   Variables   >


Thursday, March 18, 2010
                                                                     35




      Example: CQL to Brooklet
                           select Sum(shares) from trades
                            where trades.ticker = “FNM”
             CQL

        Brooklet
                                                   $total




             trades                   FNM-trades            volume

                             Filter                Sum


Thursday, March 18, 2010
                                                                            35




      Example: CQL to Brooklet
                            select Sum(shares) from trades
                             where trades.ticker = “FNM”
             CQL

        Brooklet
                               Make
                                                          $total
                           queues explicit
                             and 1-to-1
             trades                          FNM-trades            volume

                               Filter                     Sum


Thursday, March 18, 2010
                                                                                    35




      Example: CQL to Brooklet
                            select Sum(shares) from trades
                             where trades.ticker = “FNM”
             CQL                                                       Make
                                                                   state explicit
        Brooklet
                               Make
                                                          $total
                           queues explicit
                             and 1-to-1
             trades                          FNM-trades               volume

                               Filter                     Sum


Thursday, March 18, 2010
                                                                     36




      Example: CQL to Brooklet
                           select Sum(shares) from trades
                            where trades.ticker = “FNM”
             CQL

        Brooklet
                                                   $total




             trades                   FNM-trades            volume

                             Filter                  ∑




Thursday, March 18, 2010
                                                                                 36




      Example: CQL to Brooklet
                           select Sum(shares) from trades
                            where trades.ticker = “FNM” Statically
             CQL                                                 bind the original
                            Dynamically                              function
        Brooklet
                           adapt runtime
                                                        $total
                            arguments

             trades                        FNM-trades                  volume

                              Filter                      ∑




Thursday, March 18, 2010
                                                                           37


      Translation
      Correctness Theorem
    CQL/StreamIt Input                 execute      CQL/StreamIt Output

                           translate                  translate


                  Brooklet Input       execute      Brooklet Output


            Results under CQL and StreamIt semantics are the same as the
            results under Brooklet semantics after translation

            First formal semantics for Sawzall



Thursday, March 18, 2010
                                                    38




      Optimizations
       Demonstrating Brooklet’s utility
       by realizing three essential optimizations




Thursday, March 18, 2010
                                                                  39


      Operator Fusion:
      Eliminate Queueing Delays


           before

             after




                             Look for connected operators,
                           whose state isn’t used anywhere else
Thursday, March 18, 2010
                                                          40


      Operator Fission:
      Process More Data in Parallel


           before

             after



                              Split              Join




                           Look for stateless operators
Thursday, March 18, 2010
                                                                  41


      Operator Reordering:
      Filter Data Early
                                               Filter


           before

             after


                                 Filter



                      Look for operators whose read/write sets
                       don’t overlap [Ghelli et al., SIGMOD 08]
Thursday, March 18, 2010
                           42




      Outlook




Thursday, March 18, 2010
                                                                           42




      Outlook
            More optimizations

                  dynamic operator placement, load balancing,
                  subquery sharing, eliminating spurious synchronization




Thursday, March 18, 2010
                                                                           42




      Outlook
            More optimizations

                  dynamic operator placement, load balancing,
                  subquery sharing, eliminating spurious synchronization

            Richer extended calculus

                  Types, verify progress, time constraints




Thursday, March 18, 2010
                                                                                           42




      Outlook
            More optimizations

                  dynamic operator placement, load balancing,
                  subquery sharing, eliminating spurious synchronization

            Richer extended calculus

                  Types, verify progress, time constraints

            Common execution platform

                  Practical challenges: data types, library of operators, serialization,
                  process management, error handling



Thursday, March 18, 2010
                                                                                       43




      Conclusions
            Streaming is everywhere

                  Media, finance, web applications

            Need a calculus to understand (distributed) implementations

                  Minimal, non-deterministic, makes state and communication explicit

            Provide a formal and practical foundation for stream programming

                  Mappings from CQL, StreamIt, and Sawzall

                  Formalizing of Fusion, Fission, and Reordering



Thursday, March 18, 2010
                                               44




                  http://cs.nyu.edu/brooklet




Thursday, March 18, 2010
                           45




Thursday, March 18, 2010
                                      cr                                                                            e∈T P                                                                 Tuple
                                                     RName                             Relation name                σ∈Σ = bag(T P)                           Instantaneous relation
        CQL syntax:                                                                           CQL domains:
                 Pc ::= Pcr | Pcs
                                                  | S2R(Pcs )
                                                                     CQL program
                                                                                 Stream to relation
                                                                                                τ ∈T
                                                                                                                    r∈R = T → Σ                               Time-varying relation
                                                                                                                                                                     Time
                                                                                                                                                                                                         46
                Pcr ::=                           | R2R(Pcr )  (Relation query)Relation to relation                 s∈S = bag(T P ×T ) Time-varying stream
                                      Pcs
                              RName ::=                            Relation name      (Stream e∈T P
                                                                                                query)             . . . . . . . . . . . . . . . . . . . . . . . . .Tuple. . . . . . . . . . . . . . .
                                                                                                                                                                     .....
                           | S2R(Pcs )               SName   Stream to relation          Stream name bag(T P) operator signatures:
                                                                                                σ∈Σ =              CQL Instantaneous relation
                                                  | R2S (Pcr )                  Relation to stream T → Σ
                                                                                                r∈R =                                        S2R : S relation
                                                                                                                                   Time-varying × T → Σ
                           | R2R(Pcr )                     Relation to relation
                                      RName | SName ::= id                                Inputs∈S. = . . . . . . .P.×T . . .Time-varying. . . . .→.Σ
                                                                                              . . name
                                                                                                   ... ..
                                                                                                           bag(T
                                                                                                                      . ...
                                                                                                                                )
                                                                                                                                      . . . .R2S . : .Σ. × Σ . . .
                                                                                                                                              .... . .
                                                                                                                                                                  stream
                Pcs ::=                                            (Stream query)
                              SName | R2R | R2S ::= id name
                                      S2R                              Stream         Operator name
                                                                                              CQL operator signatures:                       R2R : Σ n → Σ
                                                                                                                   .............................................


    CQL Translation Rules
                           | CQL example: Bargain finder.
                              R2S (Pcr )                     Relation to stream                                    CQL S × T → wrapper signatures:
                                                                                                                   S2R : operator Σ
                RName IStream(BargainJoin(Now(quotes), history))
                               | SName ::= id                             Input name                               R2S : Σ × Σ → Σ
                                                                                                                     S2R : (Σ × T ) × {1} × S → (Σ × T ) × S
                S2R | R2R | R2S ::= id                             Operator name p                                 R2R : Σ n → Σ
                             CQL program translation: [[ Fc , Pc ]]c = .￿Fb , Pb.￿ . . . . .R2S . :. . . . .×. T.) . . .{1}.×. Σ .→ .(Σ × T ) × Σ
                                                                                              . ...... .              . . . . (Σ . . . × . . . . . . . .
        CQL example:, Bargain ]]p = ∅, outputSName;inputSName;•
                             [[ Fc SName c           finder.                                   CQL operator wrapper signatures: × (2Σ×T )n
                                                                                                                     R2R : (Σ × T ) × {1 . . . n}
        IStream(BargainJoin(Now(quotes), history))                                                  p
                                                                                               (Tc -SName) T ) × {1} × S → (Σ × T(Σ × T ) × (2Σ×T )n
                                                                                                S2R : (Σ ×                                                  → )×S
                                                                            p
        CQL program translation: [[ Fc , Pc ]]c = ￿Fb , Pb ￿                                    R2S : (Σ T ) × operator (Σ × T ) ×
                             [[ p c , RName ]]p = ∅, outputRName;inputRName;• × CQL {1} × Σ → wrappers: Σ
                                F                     c
        [[ Fc , SName ]]c = ∅, outputSName;inputSName;•                                         R2R : (Σ ×
                                                                                              (Tp -RName) T￿) × {1 . . . n} × (2q       σ, τ = d Σ×T )s = dv       n
                                                                               (Tp -SName)
                                                                                  c
                                                                                                   c
                                                                                                                   s = s ∪ {￿e, τ ￿ T ) × (2Σ×T )σ ￿ = f (s￿ , τ )
                                                                                                                                 → (Σ × : e ∈ σ}                         n
                                                                                                   p                                                                         ￿         ￿
                                p Fb , output qo ; input q; op = [[ Fc , Pcs ]]c              CQL operator wrapS2R(f )(dq , _, dv ) = ￿σ , τ ￿, s
                                                                                                                         wrappers:
        [[ Fc , RName ]]c = ∅, outputRName;inputRName;•
                                              ￿
                                            qo = freshId()                     = freshId()
                                                                             v (Tp -RName)                     σ, τ = dq                s = dv                                     (Wc -S2R)
                                                                                                 ￿
                                           ￿                                     c
                                         Fb = [S2R ￿→ wrapS2R(Fc (S2R))]Fb = s ∪ {￿e, τ ￿ : e ∈ σ}
                                                                                               s                                                σ ￿ = f (s￿ , τ )
                                            op ￿ q; op =￿ , Fc P S2R(q
               Fb , output qo ; input = op, (qo [[ v) ,←cs ]]p o , v);           c
                                                                                                                   σ, τ q , d                 σ ￿ = τ ￿,
                                                                                                     wrapS2R(f )(d= _,qdv ) = ￿σ ￿ , dv s￿ σ = f (σ, σ )
                                                                                                                                                                       ￿￿                ￿
                         ￿
                       qo =[[ freshId() cs )v p = F ￿ , output q ￿ ; input q; op ￿
                                Fc , S2R(P                 =
                                                        ]]c freshId()
                                                                    b              o                                   wrapR2S(f )(dq , _, dv )c= ￿σ ￿￿ , τ ￿, σ
                                                                                                                                                              (W -S2R)
                      ￿
                   Fb = [S2R ￿→ wrapS2R(Fc (S2R))]Fb                                                    p
                                                                                                    (Tc -S2R) ￿                                                                    (Wc -R2S)
                           ￿               ￿                                                                                                  ￿￿                   ￿
                       op = op, (qo , v) ← S2R(qo , v);                                        σ, τ = dq            σ = dv                  σ = f (σ, σ )
                                                                                                   p                                             ￿
                                    p
                                    c   = Fb , output input q; q; op
         [[ Fc , S2R(Pcs ) ]]Fb , output qo ; qo ; input op = [[￿Fc , Pcr ]]c
                                              ￿                ￿
                                                                                                   wrapR2S(f )(dq=_, dv ) = di =, di ∪ {￿σ, τ ￿}
                                                                                                                   σ, τ , dq                   ￿σ ￿￿ τ ￿,￿σ
                                              ￿
                                            qo = freshId()                   v = freshId()
                                                                                  (Tp -S2R)                               ∀j ￿= i ∈ 1 . . . n : d(Wc -R2S)    j = dj
                                                                                      c
                                          ￿
                                        Fb = [R2S ￿→ wrapR2S (Fc (R2S ))]Fb                                           ∃j ∈ 1 . . . n : ￿σ : ￿σ, τ ￿ ∈ dj
                                                 ￿               ￿
                                            op q; op = [[ Fc ← R2S               p
               Fb , output qo ; input = op, (qo , v) , Pcr ]]c (qo , v); σ, τ = dq                                  d￿ = di ∪ {￿σ, τ ￿}
                                                                                                                      i                                                   ￿
                         ￿                                                                                            wrapR2R(f )(dq , i, d) = •, d
                                                                                                                                   ￿
                                                                                                     ∀j
                                Fc , R2S (Pcr ) ]]p = Fb , output qo ; input q; op ￿￿= i ∈ 1 . . . n : dj = dj
                       qo =[[freshId()                 v = freshId()
                                                          c
                                                                     ￿              ￿
                                                                                                                                                                          (Wc -R2R-Wait)
                     ￿
                  Fb = [R2S ￿→ wrapR2S (Fc (R2S ))]Fb                                            ∃j ∈ -R2S) : ￿σ : ￿σ, τ ￿ ∈ dj
                                                                                                    (Tc p1 . . . n
                                           ￿
                       op ￿ = op, (qo , v) ← R2S (qo , v);                                       wrapR2R(f )(dq ,τ = dq •, d di = di ∪ {￿σ, τ ￿}
                                                                                                                        σ, i, d) =                 ￿ ￿

         [[ Fc , R2S (Pcr ) ]]p Fb ,Fb , output ; o ; input q; op ￿ [[ Fc , Pcr ]]p
                                               ￿                ￿
                                     c = output qo q input q; op =                                   c                        ∀j ￿= i ∈ 1(W. n : d￿ = dj
                                                                                                                                                    . . c -R2R-Wait)
                                                                                                                                                                   j
                                                           ￿                          p￿
                                 n = |Pcr |               qo = freshId()          (Tc -R2S)1 , . . . , q n
                                                                                       q =q                               ∀j ∈ 1 . . . n : σj = aux (dj , τ )
                              ∀i ∈ 1 . . . n : vi = freshId()                          ￿                  =
                                                                                                   σ, ,τop dq
                                                                                   op = op 1 , . . .                    d￿ = di ∪ {￿σ, τ ￿}
                                                                                                                          i                                                        ￿
                                                                                   p                       n       wrapR2R(f )(dd , i, d) = ￿f (σ), τ ￿, d
                                                                                                                                        ￿        q
                                        Fb = [R2R op = [[ Fc , Pcr c (R2R))](∪Fb ) ∀j ￿= i ∈ 1 . . . n : dj = j
                 Fb , output qo ;￿ input q; ￿→ wrapR2R(F]]c
                                                                                                                                                                        (Wc -R2R-Ready)
            n = |Pcr |                ￿
                                    qo = freshId() ￿ , (q ￿￿, = q← .R2R(qo , v); ∀j ∈ 1 . . . n : σj = aux (dj , τ )
                                             op ￿￿ = op             q v) 1 , . . , q n
                                                                     o
         ∀i ∈ 1 . . . n : vi = freshId()                       op ￿ = op 1 , . . . , ￿ n
                                                                                      op                                                                     ￿
                    ￿            [[ Fc , R2R(Pcr ) ]]p = Fb , output qo ;input wrapR2R(f )(dq , i, d) = ￿ ∈ d τ ￿, d
                                                             c
                                                                        ￿
                                                                                                q ￿ ;op ￿￿                    ￿σ, τ ￿f (σ),
                                                                                                                                                                            (Wc -R2R-Aux)
                 Fb = [R2R ￿→ wrapR2R(Fc (R2R))](∪Fb )                                                 p
                                                                                                    (Tc -R2R)                                  (W
                                                                                                                          aux (d, τ ) = σc -R2R-Ready)
                                               ￿
                        op ￿￿ = op ￿ , (qo , v) ← R2R(qo , v);
            [[ Fc , R2R(Pcr ) ]]p = Fb , output qo ;input q ￿Fig. 2. CQL semantics on Brooklet.
                                                   ￿               ￿
                                                                                  ;op ￿￿                ￿σ, τ ￿ ∈ d
                                        c                                                                                                           (Wc -R2R-Aux)
                                                                                  (Tp -R2R)
                                                                                      c
                                                                                                     aux (d, τ ) = σ
                             CQL, the Continuous Query Language, is a member of the StreamSQL family
                                        Fig. 2. CQL semantics on Brooklet.
                         of languages. StreamSQL gives developers who are familiar with SQL’s select-
                 CQL, from-where syntax an incremental member of the StreamSQL family
Thursday, March 18, 2010 the Continuous Query Language, is alearning path to stream programming. This
    (q1, q2, q3, $sc)                         ←               Split(q, $sc);       47
    (q4)                                      ←               wrapMap-LatLong(q1);
    (q5)                                      ←               wrapMap-LatLong(q2);
                                              ←
      Operator
    (q6)
    (out, $v4, $v5, $v6, $jc)           Fission
                                              ←
                                                              wrapMap-LatLong(q3);
                                                              Join(q4, q5, q6, $v4, $v
The following rule describes how to create the new program
of the parallelized operator.
                             op = (qout ) ← f (qin );
                                                                 ￿
       ∀i ∈ 1 . . . n : qi = freshId ()        ∀i ∈ 1 . . . n : qi = freshId ()
                     ￿                                               p
                  Fb , op s = [[ ∅, split roundrobin, q, qin ]]s
                                                 ￿
                       ∀i ∈ 1 . . . n : op i = (qi ) ← f (qi );
                    ￿￿                                             ￿ p
                 Fb , op j = [[ ∅, join roundrobin, qout , q ]]s
                                         N                ￿        ￿￿
                           ￿Fb , op￿   −→split   ￿Fb ∪   Fb   ∪   Fb , op s   op op j ￿
4
    At least in the mathematical sense; in systems, floating poi
    always commute.
Thursday, March 18, 2010

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:5
posted:9/30/2011
language:English
pages:77