Institute of Network and Information Systems

Document Sample
Institute of Network and Information Systems Powered By Docstoc
					   Reputation Studies – A
 Grassroots Approach and its
Implications for e-Science and
      Grid Computing

                    Li Xiaoming
              Peking University, China
                 December 13,2007
  (3rd IEEE e-Science Conference, Bangalore, India)
Outline
 Reputation in general
 Reputation issues in grid-like environment
 Log driven simulation – a grassroots approach
    Maze: set the stage
    Disreputable behavior analysis
    Reputation algorithms and their evaluations
 Summary and   remarks


                Institute of Network Computing and Information Systems
   Reputation in general
     Maslow’s          hierarchy of needs
         A Theory of Human Motivation, Psychological Review, 1943

Reputation is
somewhere at level
four.
“The esteem
needs. … we have
what we may call the
desire for reputation
or prestige,
recognition,
attention, importance
or appreciation.”

                           Institute of Network Computing and Information Systems
People are interested in reputation
study from various avenues:
   “Manifesto for the Reputation Society”, Hassan
    Masum and Yi-Cheng Zhang, FirstMonday, 2004
      A call by physicists on reputation systems in the information
       age.
 “A Quantitative Comparison of Reputation Systems
  in the Grid”, Jason Sonnek and Jon Weissman, Grid
  Computing Workshop 2005
 “Taxonomy of Trust: Categorizing P2P reputation
  systems”, Sergio Marti and Hector Garcia-Molina,
  Computer Networks 50, 2006
…
                       Institute of Network Computing and Information Systems
Reputation systems every where
  Academic community –               reference letters,
   paper citations, …
  e-commerce: Amazon, eBay, …




             Institute of Network Computing and Information Systems
Seller’s reputation in eBay




 The service provides means for people to rate
 each other based on transactions and provides
 a reputation profile for each seller.
               Institute of Network Computing and Information Systems
http://www.knowsky.com/tools/pr/index.asp


   WWW -- PageRank




                              Institute of # inlinks and the importance of the
The importance of a page is defined by Network Computing and Information Systems link sources
What we get from these examples
   In general, reputation is a concept that may
    be applied to any type of objects of interest,
    not only human beings, but also scientific
    papers, goods, services, etc.
   Reputation – a belief in the object’s quality,
    usefulness, capabilities, honesty and
    reliability, etc., based on recommendations.
       Forms of recommendations: reference, citation,
        hyperlink, vote, willingness to deal with, ..., etc.

                     Institute of Network Computing and Information Systems
“Calculated reputation” – a
phenomena of modern society
    Conceptually (ideally), reputation is a belief,
     of a qualitative nature
    In real life of our modern society, it is often
     approximated quantitatively -- not only
     quantitatively, but also as a total order in
     many cases .
    Reputation system is responsible
    1. for producing the quantitative reputation as realistic
       as possible
    2. for protecting itself from being manipulated (attacked)
                    Institute of Network Computing and Information Systems
Reputation issues in
grid-like environment



        Institute of Network Computing and Information Systems
   Over 30 universities voluntarily contribute vod
A perspective on grid space
   servers and maintain them on their campuses.
   The system hooks them together, forming an
   overlay network, has been running for four
   years.




                    Institute of Network Computing and Information Systems
A generic model for cyber-reputation
 Adapted from the
  GCW 2005 paper,
  works for p2p as well
  as grid scenarios
 The variations result
  when the steps are
  explicit or implicit,
  automatic or manual,
  and the RS centralized
  or decentralized, …

                  Institute of Network Computing and Information Systems
Issues of concern in reputation studies
 How to compute a reasonable reputation score
  when participants behave “normal”.
    Often, the algorithm needs to reflect some
     organization level policy, which encourages or
     discourages some behaviors even if they are both
     normal in “normal” sense.
 How  to make the reputation system robust
  against “abnormal” behaviors (attacks)
    Free riders, badmouthers, ballot-stuffers, colluders,
     malicious providers, whitewashers, Sybil attacks, …

                  Institute of Network Computing and Information Systems
The core message from the
above:
   Reputation is mainly a function of behaviors
      Behaviors of a service, or the person/organization behind the
       service
      Behaviors of entities using the service
      Behaviors of entities “hired” by some services
 As such, user/entity behavior analysis/understanding
  should be a central issue in reputation studies, at least
  as important as algorithm design
 Moreover, the quality of an algorithm is ideally
  evaluated through real behavioral data

                      Institute of Network Computing and Information Systems
Three kinds of data used in
research practices
  Synthetic data
     Generated from some model instrumented by a
      few parameters
     EigenTrust, TrustGuard, PowerTrust, …, and the
      GCW paper
  System   operational log data
     Credence, Maze, …
  User   (human) feedbacks (votes)
     eBay, …

                Institute of Network Computing and Information Systems
Reputation studies: a log driven approach

 Maze:   set the stage
    The most popular P2P file sharing system run on
     CERNET (China Education and Research
     Network): over 2 million registered users since
     2004, indexing over 200 million files, generating
     about 20TB traffic every day, …
 Disreputable behavior analysis
    Free riders, colluders, and file pollutions, …
 Reputation    algorithms and their evaluations
    LIP and EigenTrust

                  Institute of Network Computing and Information Systems
                                              Credits -
                                               Points


Maze Interface Search
        Tianwang File

                                                                          Seed
                                                                        Mechanism




          IM
                                                                  Multi-Source
                                                                  Downloading

                                                             Downloading List
                                                          (queuing, access NAT)
                                   Browser


 Social
Network        Institute of Network Computing and Information Systems
Maze induces two useful networks

 Download network
    There is a directed edge from peer A to peer
     B, iff B has downloaded file (s) from A
 Friendship Network
    There is a directed edge from peer A to peer
     B, iff A has chosen B as his friend explicitly
     in the Maze interface.



                 Institute of Network Computing and Information Systems
                                                   • Encourage becoming a
                                                     Mazer
                                                   • Encourage upload
Award based incentive policy                       • Bent to large downloads


 New peers are initialized with 4096 points.
 Uploads: +1.5 points per MB uploaded
 Per file downloaded:
       -1.0/MB downloaded within first 100MB
       -0.7/MB per additional MB between 100MB and 400MB
       -0.4/MB between 400MB and 800MB                           Account Change(Point/Mb)
                                                                                         1




       -0.1/MB per additional MB over 800MB                                                        Upload

                                                           900M      500M          100M




   Service differentiation:                                                                 -0.1            Transfer Bytes(Mb)
                                                                      Download
                                                                                             -04

                                                                                             -0.7

                                                                                             -1



       Each peer orders download requests by
        T = requestTime – 3logP, where P is the requester’s point total.
       Users with P < 512 are limited to 200Kb/s

    This policy is published. Computing and Information Systems
                    Institute of Network
An operational abstraction and log model

                                       LOG MODEL
                                        Identification
                                               unique machine id
                                               IP address
                                               peer id
                                           Activity
                                               on/off line periods of each
                                                peer
                                               transfer starting and ending
                                                time
                                               bytes transferred
                                           Metadata
                                               periodically upload directory
                                                information to index server,
                                                including file name, creation
                                                time, size, md5 hash, etc.

             Institute of Network Computing and Information Systems
How Maze fits in the grid reputation
model ?
   0. registration
      A user installs the software, as if he claims he will provide
       disk space and network bandwidth
   1. client requests for recommendation
      Send a search query
   2. recommendation
      Maintain a list of good peers
      A list of candidate objects in order as response to request
   3. transaction activity
      File download
   4. feedback
      Implicit: file retention time, points granted to the uploader;
      Explicit: votes, comments on a forum

                        Institute of Network Computing and Information Systems
Disreputable behavior analysis
 Objective: identifyquestionable peers and
  understand the patterns of their behaviors
    Free-riders, colluders, polluters, …
 The   key in this kind of study
    How the behavioral notions are defined in terms of
     specific system operations.
    What data can be collected from the system to
     derive the conclusions about the notions
    (The whole business is a matter of mapping
     between qualitative notions and quantitative
     measurements)
                   Institute of Network Computing and Information Systems
A study on free riding (IPTPS’05)
 The   question asked
    Is there free riding behavior in Maze ?
    If yes, how is it ?
 What   do you mean by “free rider” ?
    Intuitively, download much more than upload
    Precisely, a peer is considered a free rider if its
     points (reputation) less than initial value
 What   do you mean by “how” ?
    Impact to the system, bandwidth consumption, etc
    Intentionality study (but how ?)

                  Institute of Network Computing and Information Systems
Log Data for free-riding study
(9/28~10/28, 2004)
Log duration                                   30 days
# of active users                              130,205
# of NAT users                                 51,613 (40%)
# of transferred files                         6,831,019 (avg 14.2MB/file)
# of unique transferred files                  1,588,409 (avg 4 copies/file)
Total transfer size                            97.276TB
Average transfer speed                         328 Kb/s
# of search                                    2,282,625

                      Institute of Network Computing and Information Systems
Ratio of free-riders
   free-riders occupies 82% of the Maze population




                   Institute of Network Computing and Information Systems
Understanding free-riding
 Do   free-riders behave intentionally ?
   Do they switch off their servers ? (Maze
    allows it)
   Do they hide their files ? (note the basic
    principle of a P2P system)
   Are their machines/connections too slow to
    allow others downloading ?
   Do they quit sooner ?
 These are all measurable and partially reflect some
 kinds of intentions
                 Institute of Network Computing and Information Systems
Do Free-riders Hide Their Files?


                                                                     “No”




   The average number of shared files of free-riders is 491,
    versus 281 of the server-like users.
   Free-riders also keep a good portion of interesting files in
    their directory.
                      Institute of Network Computing and Information Systems
Upload speed versus free-riding
        # of Average Transfer Speed(Kbps)

                                                       500
                                                       450
                                                       400
                                                       350
                                                       300
                                                       250
                                                       200
                                                       150
                                                       100
                                                        50
                                                         0
                                            -12   -7   -2          3         8         13   18     23
                                                             # Change of User Points(10K)


Free-riders are not
necessarily handicapped                                                          Good bandwidth necessary to
(in Maze)                                                                        earn high points (Maze capped
                                                                                 at 500kbps)


                                                       Institute of Network Computing and Information Systems
Session time is important
                                                      45000
      # of User Online time (sec)



                                                      40000
                                                      35000
                                                      30000
                                                      25000
                                                      20000
                                                      15000
                                                      10000
                                                       5000
                                                          0
                                    -15   -10   -5            0        5          10     15    20       25
                                                          # Change of User Points(10K)



    Free-riders just quit sooner (may not be
     intentional, though):
       Users with positive point changes stays 2.89 times
        longer (218 mins versus 75 mins).

                                                     Institute of Network Computing and Information Systems
A study on colluding behavior
(ICDCS’07)
   We ask
      Is there colluding behavior in Maze ?
      If yes, how is it ?
   What do you mean by “colluding behavior” ?
      Intuitively, those collective activities to take advantage of
       Maze asymmetric awarding policy and earn points without
       actual contribution (a waste of network bandwidth).
      Precisely, a set of detecting criteria are designed to identify
       suspicious colluders
           • Repetition detector: watch if a peer uploads the same file to
             the same peer many times.
           • Pair-wise detector: watch if a pair of peers have mutual
             upload close to their total uploads

                        Institute of Network Computing and Information Systems
Repetition detector
    Duplication degree = total upload Bytes / unique Bytes

                                                                 Unique data Total traffic
                                                                                             Temporal locality
                                                      Src ID       on max       on max
                                                                                             (x: date, y: upload)
                                                                     edge        edge
                                                       Alice        7.5 GB      126 GB
                                                       Bob          6.0 GB      98 GB
                                                      Cindy         1.9 GB      81 GB
                                                      David         3.1 GB      62 GB
                                                      David        10.1 GB      52 GB
                                                       Eric         7.4 GB      44 GB



 Colluders generate large amounts of traffic with repeated contents
 221,000 transaction pairs, (roughly 4.9%) contain duplicate files
 Closer look
     Strong temporal locality present
     Generally large files or directories (Alice uploaded a 3GB DVD image 29 times)
                                  Institute of Network Computing and Information Systems
Mapping the collusion topology




   100 links with the highest ratio of duplicate transfers
      Pair-wise, star-shaped, 3-party ring
                      Institute of Network Computing and Information Systems
Pair-wise detector
Pair-wise degree = sum of bidirectional traffic/ sum total uploads


                                                  Peer 1                                      Peer 2
                                                            Peer 1   Mutual upload   Peer 2
                                                 external                                     external

                                                  1.7GB     Fred     24GB23GB      Gary      5GB

                                                  23GB      Cindy    81GB27GB      Harry     0GB

                                                                     62GB126G
                                                  52GB      David                    Alice     32GB
                                                                           B




           Large amounts of mutual upload traffic compared to total
           uploads. 28,000 pairs of peers with mutual Uploads.
           Closer   look: Most colluding peers have similar IP addresses


                            Institute of Network Computing and Information Systems
What did we learn from this ?
 Collusion behavior isintuitive, but still
  application dependent. How it appears in
  computational grid service is yet to see.
    Difficult to craft universal collusion detector, but
     system logs usually provide good hints.
 (theabove study about collusion is based on
  one-month log data: Feb. to March 2005)
    161,000 active users
    32 million file transfers
    Total data traffic > 437 TB
                    Institute of Network Computing and Information Systems
A study on file pollution
(IPTPS’07)
 The   observed behavior
    People have at least two good reasons to
     purposely “pollute” files
    Polluted files are often popular (people don’t bother
     to pollute those unpopular files)
 We  ask: how do we tell if a file has been
  polluted with high probability ?
    Two detectors are designed
    A more elaborative scheme came out of the
     process
                 Institute of Network Computing and Information Systems
Motivations to “pollute” files
 Discouraging illegal          downloads
    copyright companies, such as Overpeer and
     Loudeye
    injecting damaged files into the system
 Making  use of innocent incentive policy of a
  system, and to gain undeserved advantage in
  the community
    such as some peers in Maze
    renaming files and pretending they are “new arrivals”


                  Institute of Network Computing and Information Systems
The process from victim’s view
                                                     Wendy has!




                          Server
              Wendy, can you give me
             “Mission: Impossible III”?
          Who has “Mission: Impossible III”?


   Bob                                                          Wendy

                                                     Sure!

                             Alice
            Institute of Network Computing and Information Systems
The process from victim’s view
 But    when Alice plays it, she finds that…


        Isn’t it “Matrix Reloaded” ?
   But I want “Mission: Impossible III” !
          Wendy has cheated me!




                      Alice
                        Institute of Network Computing and Information Systems
The process from attacker’s view
                                     Or, I have an unpopular file
                                      “Matrix Reloaded.rmvb”.
                                     But I know “Mission: Impossibleme,
                                           No one downloads file from III”
                                                 So I have no point,
                                           my is very to be Mission
                                    Can I pretend it popular now,
                                               download have a limited
                                         because Mm… speed is popular file.
                                                   I do not
                                              Impossible? and share it?
                                                  download
                                       should I always in the last position
                                        and I’m
                                     I can change the filename!
                                            of the downloading queue.




      Matrix Reloaded
Mission: Impossible III.rmvb
                                                                    Wendy

                Institute of Network Computing and Information Systems
The process from attacker’s view

                                                  Mm, I have got some points.
                                                 “Mission: Impossible III.rmvb .
                                          I haveBut how can I attract more users”to
                                                       download from me?


                                           Ok, I know.
                    Server
             “Spider-Man 3” is going to be played next
                 month, so there is no real file now.
                     Wendy, filename to this
                  If I change can you give meone,
                    “Mission: Impossible III”?
                all the users who want to download
                                Transfer
               this movie will download it from me!
                     “Mission: Impossible III.rmvb”
                                         Sure!
     Alice                                                          Wendy
                   Institute of Network Computing and Information Systems
Problem observation
   Popular files are often polluted in Maze
      users spend more time to find real files.
      waste lots of network resource for repeated download.
      in frustration, users give up peer-to-peer system finally.

 Resource title   # all versions     # fake versions pollution percentage


      T1             1606                   779                     48.5%
      T2              602                   191                     31.7%
      T3              491                    82                     16.7%
      T4              203                    52                     25.6%

                         Institute of Network Computing and Information Systems
How do we tell ?
 Detector 1:   relative timing
    For two files of the same name but different
     md5 values, the one with later creation time
     is suspicious to be a fake file.
 Detector 2:   birth-day verification
    If a file’s creation time is earlier than the
     public announcement time (in real life) of
     the implied object, the file is suspicious to
     be fake.

                 Institute of Network Computing and Information Systems
What other hints log data may give us ?
 File   retention time !
    Fake files are usually deleted quicker.
 Two assumptions for an more elaborative
  scheme
    assumption 1: file retention time is proportional to
     its value
    assumption 2: file popularity (number of
     downloads) is proportional to the probability of
     being polluted
 They  give rise to a file ranking scheme in
  peer to peer system !
                  Institute of Network Computing and Information Systems
LIP: a ranking scheme combining
lifetime and popularity
   Classification of files
     zone 1
     • High popularity
     • Long lifetime
     • more likely to be real files
     zone 4
     • High popularity
     • Short lifetime
     • more likely to be fake files
     zone 2&3 are to be treated separately
     • See our paper in IPTPS’07

                        Institute of Network Computing and Information Systems
Reputation algorithms and
their evaluations
 In this part, we’ll show how the system logs
  can also be used to evaluate algorithms, reveal
  specific weakness, and give hints to improve
  them, in addition to the user behavior analysis.
    Log driven evaluation of LIP
    We also evaluate EigenTrust, one of the
     most popular algorithms to cope with
     collusions. Surprisingly, despite its popularity,
     it has not been deployed in large scale
     systems.
                 Institute of Network Computing and Information Systems
LIP (IPTPS 2007 and beyond)
 As we’ve just indicated, LIP results in a file
  recommendation system
    In our IPTPS’07 paper, only “synthetic” data
     were used in evaluation
A   deployment was done after IPTPS’07
    Has run 80 days, recommended 114,257
     resources to 17,487 users, which results in 5206
     downloads involving 2308 users.
 How   do we tell if it is effective ?
    Log driven simulation !
                 Institute of Network Computing and Information Systems
Retention time tells the story
    For each user, sort the relative ranks in terms of
     retention time for all his file, and see if the
     recommended files appear to have higher ranks.
                                                 0.6
                                                                             ordinary
                                                 0.5                         recommended

                                                 0.4




                                          CCDF
                                                 0.3

                                                 0.2

                                                 0.1

                                                  0
                                                       0   0.2   0.4   0.6   0.8   1       1.2
                                                                   life rank




  (x,y): y percentage of files whose life rank are higher than x
                    Institute of Network Computing and Information Systems
Evaluate EigenTrust (CTS 2006)
 To our knowledge, EigenTrust algorithm,
  though elegant, has not been tested in real
  peer-to-peer system as of 2006.
 We ask:
    How is EigenTrust for real system like
     Maze ?
    How can we improve it if it is not good
     enough ?


               Institute of Network Computing and Information Systems
EigenTrust Algorithm
   S.D. Kamvar, et al, “The      ( 0) 
    EigenTrust Algorithm for     t  p;
    Reputation Management in P2P
    Networks”, Procs of WWW, May repeat
    2003                            ( k 1)                       (k )
                                                t          C t ;  T
 C: initial trust matrix                        ( k 1)                (k ) 
 p: pre-trusted peers                          t           (1  a )t  ap;
 a: balancing parameter                                   ( k 1)  ( k )
                                                 t               t
 When converged, elements
  of t represent reputation                  until              ;
  values of corresponding
  peers
                    Institute of Network Computing and Information Systems
How do we map the algorithm to a
real system like Maze ?
 Namely,  what is proper C, p, a, especially C ?
 Recall, Maze induces two networks
    Download network: there is a directed
     edge from peer A to peer B, iff B has
     downloaded file(s) from A
    Friendship Network: there is a directed
     edge from peer A to peer B, iff A has
     chosen B as his friend explicitly in the
     Maze interface.

                Institute of Network Computing and Information Systems
The Mapping from Maze to EigenTrust

 Now,  we construct the matrix C, pre-trusted
  peer vector p, and balancing parameter a.
    c(i,j): proportional to the total downloading of peer i from peer j
     during the log period, with normalization
    p: based on the information from Maze forum, we select 10
     peers that we are confident that they can act as the pre-trusted
     peers.
    set a=0.15.
 Experiment is        done with one month of log data
    2005.2.19 – 2005.3.24
    182,223 peers involved
    495TB data exchanged among them

                      Institute of Network Computing and Information Systems
The EigenTrust Values of the Peers

                                                                  The points in
                                                                  the Region-L
                                                                  imply that the
                                                                  peers are under
                                                                  evaluated for
                                                                  their
                                                                  contributions

    Overall, the more uploads a peer has, the higher score this
     peer will get.
    The peers are spread in two noticeable bands (regions)
    A closer look reveals three problems with EigenTrust.
                      Institute of Network Computing and Information Systems
EigenTrust is unfair for local distributors




    A peer is called a local distributor if its upload contribution
     is mainly to the peers within the same organizational
     domain
    We identified a peer, Wayne, who uploaded a lot, but found
     in Region-L. EigenTrust is unfair for Wayne-like peers

                       Institute of Network Computing and Information Systems
Pre-trusted peers may help colluders
unexpectedly
 Total upload     Total     Ted download                       Ted’s total
    by Larry      collusion   from Larry                        downloads

    29.7GB          29GB               734KB                       124M

   We identified a pre-trusted peer, Ted, and a colluder Larry,
    investigate the traffics involved them in detail. We realize:
      A pre-trusted peer normally download less and as a result it
       will give high trust value cij to those he downloaded from
      This implies that global ranking doesn’t work well in this
       instance.


                      Institute of Network Computing and Information Systems
The algorithm is sensitive to the
distribution of pre-trusted Peers
   Three scenarios were examined
      The original one, manually choose 10 peers who we trust.
      Choose m most active peers who have uploading more than
       others (top-m).
      Choose m peers randomly from the whole peer set
   Run EigenTrust with the same batch of logs for the different
    cases and see how eigenvalues are changed in terms of ranks

                           M = 100 M = 1000                M = 2000             M = 3000

Average distance of rank
     changes relative to      801            981              1097                  1165
              scenario 1


                           Institute of Network Computing and Information Systems
How can we improve ?

 The  algorithm is unfair to many local
  distributors.
 Pre-trusted peers may help colluders to elevate
  their eigenvalues unexpectedly.
 The rank of a peer’s EigenValues is very
  sensitive to the selection of pre-trusted peers.



                Institute of Network Computing and Information Systems
Improvement 1: a location-aware pre-
trusted peers selection algorithm
   Make the pre-trusted peers equally
    distributed in IP address zones. (benefit to
    the local distributors)
    1. Aggregate IP addresses into zones by using the
       WHOIS service. A zone refers to an intranet of a
       university, college or a research institution etc.
    2. Proportionally determine the number of pre-
       trusted peers that should be selected in each
       zone
    3. Select pre-trusted peers in every zone. A peer’s
       probability to be a pre-trusted peer is in proportion
       to his in-degree in download network.
                    Institute of Network Computing and Information Systems
                                                                    Improved
                                                                    Result 1




   The number of peers in Reign-Low reduces from 657 to 289.
   For those five marked local-distributors, their rank positions
    in whole peers increase 6 times in average
                       Institute of Network Computing and Information Systems
Enhance the trust matrix C with
friendship network
 Rational: the subjective friendship network
  should be more accurately indicate the trust
  relationship other than the objective
  download network between pre-trusted peers
  and other peers. (we want to punish leg-
  huggers)
    Therefore, we assume the pre-trusted peers
     should only trust their friends, do not trust those
     who download from him but not friends.
    combine these two networks to construct the trust
     matrix C.

                 Institute of Network Computing and Information Systems
                                                                Improved
                                                                Result 2



   Some colluders are clearly punished.
   Cannot completely avoid the impact that the pre-trusted
    peers’ unintentional actions bring to us. (maybe a pre-trusted
    peer adds a colluder as his friend, because he has high points)
                      Institute of Network Computing and Information Systems
Summary and remarks
 Parallel to security in some sense, reputation is an
  inevitable element in cyber activities, e-science and
  grid computing have no exception.
 As long as reputation is calculated and associated
  with some kinds of profit, temptations are inevitable.
  Some suspicious behaviors have shown concretely in
  a real system.
 As opposed to model driven approach from the top,
  log driven simulation can help algorithm evaluation
  and design in some convincing way, as well as reveal
  user behavior specifics and patterns, otherwise
  impossible.
                  Institute of Network Computing and Information Systems
Summary and remarks
 Mapping from intuitive reputation concepts to
  semantically simple log data is the main theme in
  log driven simulation.
 With a system running, it is no difficulty to collect
  millions of log data items, but cleaning and
  verifying them (whether they constitute a valid
  sample) is a scientific challenge in the grassroots
  approach. Little has been done in this regard.
  Perhaps a combination of model driven and log
  driven approaches is the future.

                   Institute of Network Computing and Information Systems
Questions & Comments


 Contact:   lxm@pku.edu.cn

               Institute of Network Computing and Information Systems
Institute of Network Computing and Information Systems

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:2
posted:9/17/2012
language:English
pages:64